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FOREWORD 


This  volume  is  one  of  nine  individually  bound  volumes  that  constitute 
the  Phase  IX  Final  Report  "Study  of  Embedded  Computer  Systems  Support" 
for  Contract  F33600-79-C-0540.  The  efforts  and  analyses  reported  In 
these  volumes  were  sponsored  by  AFLC/LOEC  and  cover  a  reporting 
period  from  September  1979  through  September  1980. 

The  nine  volumes  are 
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1.  AUTOMATIC  TEST  SYSTEMS 


1.1  INTRODUCTION 

Automatic  testing  began  in  the  1950's  as  a  result  of  the  advent  of 
equipment  that  required  high-speed  device  testing.  The  sequence  of 
simple  tests  were  so  numerous  and  frequent  that  an  electrical  means 
was  devised  to  administer  the  tests.  In  subsequent  years  the  explosive 
growth  of  semiconductor  devices  created  an  additional  demand  for  auto¬ 
matic  test  equipment.  The  semiconductor  devices  were  physically 
mounted  on  boards  known  as  circuit  boards.  As  the  growth  in  circuit 
boards  paralleled  the  growth  of  test  devices,  the  technician  found  him¬ 
self  unable  to  adequately  test  with  an  oscilloscope,  thus  subassembly 
testing  was  born. 

Since  those  times  the  electronic  products  themselves  have  magni¬ 
fied  in  complexity  and  number  and  automatic  testing  has  been  forced  to 
include  fault  diagnosis  which  has  increased  the  complexity  of  the  ATE. 
As  any  new  break  through  in  electronics  occurs  (such  as  large  scale 
integration),  additional  demands  are  created  for  new  or  revised  ATE. 

It  is  apparent  from  this  description  that  the  ATE  arena  has  been  and 
remains  in  the  midst  of  a  fierce  evolution. 

A s  the  needs  for  ATE  intensified,  AFLC  was  faced  with  the 
dilemma  of  reduced  numbers  of  personnel  and  a  threat  of  less  training 
for  support  personnel.  The  concept  of  using  automatic  testing,  where 
practical,  to  replace  added  manning  was  conceived  and  implemented. 
Ideally,  additional  machine  controlled  tests  would  require  fewer 
trained  personnel  to  accomplish  the  required  tests.  Accordingly,  the 
Air  Force  has  supported  expanded  usage  of  automatic  test  methods  in 
spite  of  the  rapid  technological  evolution  of  automatic  test  systems. 
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Support  for  Automatic  Test  Systems  has  progressed  to  an  equally 
dynamic  state  and  as  the  criticality  of  avionics  to  the  aircraft's  mission 
becomes  more  acute  the  importance  of  ATE  increases.  AFLC  is  aware 
of  the  dynamic  situation  and  is  taking  steps  to  improve  the  support  of 
ATE  and  the  avionics  which  utilize  ATE.  This  volume  describes  the 
current  baseline  for  ATE  support  throughout  the  command. 

1.2  DEFINITIONS 

Several  definitions  are  offered  here  so  that  a  common  basis  of 
understanding  is  established  for  discussion  of  ATE  support.  Those 
definitions  marked  with  an  asterisk  were  extracted  from  the  IEEE 
Standard  Dictionary  of  Electrical  and  Electronics  Terms. 

1.2.1  Automatic  Test  Equipment 

Electronic  devices  capable  of  automatically  or  semiautomatically 
generating  and  independently  furnishing  programmed  stimuli;  measuring 
selected  parameters  of  an  electronic,  mechanical,  or  electro -mechanical 
item  being  tested;  and  making  a  comparison  to  accept  or  reject  the 
measured  values  in  accordance  with  predetermined  limits.  ATE  may 
also  include  independently  configured  automatic  or  semiautomatic 
devices  which  are  capable  of  detecting,  measuring,  and  evaluating 
electrical,  electronic,  or  electro -mechanical  characteristics  of  system/ 
equipment.  ATE  normally  operates  by  use  of  previously  prepared  test 
software  recorded  on  punched  tape,  card  decks,  magnetic  tapes,  disc 
pack,  or  other  storage  media. 

1.2.2  Automatic  Test  System 

Those  equipment,  software,  and  data  items  required  to  operate 
and  maintain  ATE  and  test  software  used  thereon.  This  system  includes 
test  equipment,  interface  test  adapters,  test  software,  compilers, 
programming  information,  tester  data,  but  not  off-line  Automatic  Data 
Processing  Equipment  (ADPE)  used  in  support  of  test  software. 


1.2.3  ATE  Control  Software 

Software  used  during  execution  of  a  test  program  which  controls 
the  non-testing  operations  of  the  ATE.  This  software  is  used  to  execute 
a  test  procedure,  but  does  not  contain  any  of  the  stimuli  or  measure¬ 
ment  parameters  used  in  testing  the  Unit  Under  Test  (UUT).  Where 
test  software  and  control  software  are  combined  in  one  inseparable 
program,  that  program  will  be  treated  as  test  software,  not  control 
software. 

1.2.4  ATE  Support  Software 

Computer  programs  which  aid  in  preparing,  analyzing,  and 
maintaining  test  software.  This  software  includes  ATE  compilers, 
translation/analysis  programs,  and  punch/print  programs.  This 
software  is  never  used  during  the  execution  of  a  test  program  on  a 
tester.  Support  software  may  exist  off-line  to  the  ATE  or  may  be 
resident  (on-line)  in  the  ATE. 

1.2.5  Built-In-Test* 

A  test  approach  using  built-in-test  equipment  or  self-test  hard¬ 
ware  or  software  to  test  all  or  part  of  the  unit  under  test. 

1.2.6  Interface  Teat  Adapter 

The  equipment  necessary  to  provide  physical  and/or  electrical 
compatibility  between  the  UUT  and  the  ATE.  The  Interface  Test 
Adapter  (ITA)  normally  consists  of  cabling,  wiring,  connectors,  and 
miscellaneous  components. 

1.2.7  Self-Test* 

A  test  or  series  of  tests  performed  by  a  device  upon  itself,  which 
shows  whether  or  not  it  is  operating  within  designed  limits.  This  includes 
test  programs  on  computers  and  automatic  test  equipment  which  check 
out  their  performance  status  and  readiness. 
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1.2.8  Simulator  Teat 

A  device  or  program  used  for  test  purposes  which  simulates  a 
desired  system  or  condition  providing  proper  inputs  and  terminations 
for  the  equipment  under  test. 

* 

1.2.9  Support  Equipment 

Equipment  required  to  make  an  item,  system,  or  facility  opera¬ 
tional  in  its  environment.  This  includes  all  equipment  required  to 
maintain  and  operate  the  item,  system,  or  facility  and  the  computer 
programs  related  thereto.  (ATE  is  a  subset  of  support  equipment). 

* 

1.2.10  Test  Point 

A  convenient,  safe  access  to  a  circuit  or  system  so  that  a 
significant  quantity  can  be  measured  or  introduced  to  facilitate  main¬ 
tenance,  repair,  calibration,  alignment,  and  check  out. 

1.2.11  Test  Program  Set 

A  Test  Program  Set  (TPS)  consists  of  a  complete  software 
package,  including  test  tape  or  disc,  interface  test  adapter,  and 
test  program  instructions.  (TPS  is  sometimes  used  by  the  Air 
Force  as  an  abbreviation  for  Test  Program  Specification.  The  use 
throughout  this  document  applies  to  Test  Program  Set  only.) 

$ 

1.2.12  Test  Requirements  Document 

The  document  that  specifies  the  tests  and  test  conditions 
required  to  test  and  fault  isolate  a  unit  under  test. 
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1.2.13  Test  Requirements  Document 

A  document,  described  by  MIL -STD  1519,  which  records 
performance  and  calibration  requirements  of  testable  end  items.  ^ 

1.2.14  Test  Software 

Computer  programs  which  control  the  testing  operations  and 
procedures  (including  certification  and  fault  isolation  procedures) 
of  the  ATE.  The  program(s)  used  to  control  the  stimulus  and 
measurement  parameters  used  in  testing  the  UUT.  It  is  also 
referred  to  as  a  test  program  or  test  tape. 

,  * 

1.2.15  Test  Support  Software 

Computer  programs  used  to  prepare,  analyze,  and  maintain 
test  software.  Test  software  includes  Automatic  Test  Equipment 
(ATE)  compilers,  translation/analysis  programs,  and  punch/print 
programs,  v, 

1.2.16  Unit  Under  Test* 

Any  system,  set,  subsystem,  assembly,  subassembly,  etc., 
undergoing  testing. 

1.2.17  Unit  Under  Test  Manager 

The  organization  assigned  material  management  responsibility 
for  the  prime  equipment  subject  to  automatic  testing,  that  is,  the  UUT. 


There  are  two  definitions  of  TRD's  included  because  they  differ  in 
their  content.  The  IEEE  definition  encompasses  fault  isolation  while 
the  definition  extracted  from  AFLCR  66-37  includes  calibration 
requirements.  Use  of  the  term  TRD  in  this  volume  is  intended  to 
include  a  composite  of  both  definitions. 
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1.3  ATE  SYSTEM  IDENTIFICATION 


The  varied  ways  that  automatic  equipment  is  used  to  solve  testing 
problems  indicates  that  no  unique  general  purpose  way  to  accommodate 
all  of  these  test  requirements  exists.  In  addition,  there  is  no  centralized 
acquisition  or  management  activity  to  control  proliferations.  Each  test 
user  approaches  automatic  testing  with  his  own  particular  needs  and  tends 
to  optimize  his  specific  test  system  to  solve  his  needs.  As  a  result  the 
total  number  of  ATE  currently  within  AFLC  responsibility  is  very  large 
and  the  potential  for  even  more  systems  is  high  as  future  weapons  systems 
are  acquired. 

During  acquisition  of  a  weapon  system  AFLC  provides  assistance  to 
AFSC  in  determining  the  support  equipment  (to  include  ATE)  that  will  be 
necessary  for  life  cycle  support  of  the  weapon  system.  AFSC  is  charged 
with  the  responsibility  to  procure  the  weapon  system  and  the  support 
equipment.  Subsequent  to  Program  Management  Responsibility  Transfer 
(PMRT)  AFLC  assumes  complete  management  responsibility  for  the 
support  of  the  weapon  system.  At  that  time  the  weapon  system  normally 
has  become  operational  and  all  levels  of  testing  should  be  supportable. 

AFLC  is  involved  in  three  levels  of  testing  using  automatic  equip¬ 
ment:  organizational,  intermediate,  and  depot.  (The  combination  of 
organizational  and  intermediate  is  occasionally  referred  to  as  field  level.  ) 
A  complete  baseline  of  ATE  would  encompass  the  entire  set  of  ATE  for 
all  three  test  levels.  While  only  one  baseline  may  appear  convenient 
from  a  Headquarters  AFLC  perspective,  actual  implementation  of  only 
one  overall  testing  level  is  impractical  due  to  increased  costs  and  the  wide 
variance  between  depot  and  organizational  testing  levels.  Figure  1-1 
indicates  the  affiliation  of  the  three  testing  levels. 

Notice  that  although  ATE  applies  at  all  three  testing  levels  the 
extent  or  type  of  testing  varies  from  level  to  level.  Typically  there  is 
only  one  group  of  UUT  test  programs  and  one  test  set  for  each  avionics 


Figure  1-1.  Relationship  of  ATE  Testing  Levels  for  System  X 


system  at  the  organizational  level.  Ideally,  but  also  impractically,  the 
one  test  set  could  apply  to  more  than  one  avionics  system  which  would 
reduce  the  overall  number  of  test  sets  required,  however  this  is  not  the 
case  for  most  avionics  systems.  At  the  intermediate  level  at  least  one 
UUT  program  exists  for  each  Line  Replaceable  Unit  (LRU)  and  as  many 
test  sets  exist  as  are  required.  That  is,  if  there  are  7  LRU's  total  and 
test  set  (A)  will  test  3  of  the  SRU's  and  another  test  set  (B)  will  test  2 
LRU's  and  the  other  2  LRU's  required  unique  test  sets  then  a  total  of 
4  test  sets  are  required  with  7  U?TT  test  programs.  A  similar  example 
could  be  cited  where  individual  UUT's  require  more  than  one  test  pro¬ 
gram.  At  the  depot  level  SRU's  are  tested  for  component  failure. 

The  APG-63  radar  set  which  is  used  on  the  F-15  weapon  system 
could  be  used  as  an  example  of  avionics  subsystem  A  in  Figure  1-1. 

There  are  9  LRU's  and  over  200  SRU's  in  the  APG-63.  If  all  subsystems 
used  on  all  weapon  systems  are  totaled  for  AFLC,  then  it  is  evident 
that  vast  numbers  of  test  sets  and  test  programs  are  required  to  provide 
ATE  support. 

If  the  F-15  is  again  used  as  the  example  of  weapon  system  X  in 
Figure  1-1,  then  the  system  management  is  accomplished  by  the  F-15 
SM.  The  three  levels  of  ATE  are  managed  by  item  managers  who  are 
not  all  necessarily  collocated  with  the  SM.  The  SM  is  responsible  for 
knowing  who  the  respective  item  managers  are  for  his  system  and  for 
integrating  their  efforts  to  provide  support  for  his  system.  Management 
complexity  itensifies  when  certain  avionics  subsystems,  LRU's,  or  SRU's 
are  used  on  more  than  one  weapon  system.  Thus  an  item  manager  is  now 
supporting  multiple  SM's. 

The  total  ATE  baseline  within  the  command  is  the  composite  of  all 
UUT  programs  and  test  sets  for  all  weapon  systems  readjusted  for  any 
common  equipment  and/or  programs  that  apply  to  multiple  avionics  sub¬ 
systems,  LRU's,  or  SRU's.  Table  1-1  indicates  an  overview  of  the  ATE 
for  the  F-15  aircraft.  Note  the  multiplicity  of  testers,  vendors,  and 
users  involved  with  only  this  one  weapon  system. 
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Appendix  A  contains  a  listing  of  all  the  ATE  systems  within  SA-ALC 
responsibility.  A  count  of  the  number  of  systems  will  not  necessarily 
indicate  the  scope  of  total  support  requirements  because  each  system 
has  different  demands  upon  time  and  personnel.  ATE  software  is  also 
a  part  of  the  current  baseline.  Identification  of  all  software  programs 
is  a  requirement  of  the  Computer  Program  Identification  Number  (CPIN) 
system.  Category  85  of  the  CPIN  system  singles  out  the  computer  pro¬ 
grams  peculiar  to  test  stations  and  testers.  This  system,  managed  by 
OC-ALC,  indicates  category  85  programs  by  subcategories  of 

e  Electronic  warfare 

•  Communications 

•  Data  processing  and  display 

•  Engines 

•  Flight  controls 

•  Guidance 

•  Navigation 

•  Weapons  delivery 

•  Fire  control 

•  Electronic  and  electrical 

•  Armament  and  munitions 

•  Multiple  major  functions 

•  Hydraulic 

•  Pneumatic 

•  Pneudraulic 

•  Vacuum 

•  Surveillance/tracking 

•  General  purpose  or  supportive 

In  the  future,  interrogating  the  CPIN  system  will  indicate  the  number  of 
test  related  programs  for  any  subcategory.  The  initial  data  base  for  the 
CPIN  system  is  not  yet  complete  and  the  system  is  currently  resident 
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on  a  word  processor  type  machine.  As  a  result  the  CPIN  system  is  still 
a  manual  capability,  but  to  be  highly  useful  its  dynamic  data  base  state 
requires  a  computer  aided,  rapidly  updateable  capability.  Currently, 
the  CPIN  system  contains  reference  to  only  approximately  7200  ATE 
computer  programs. 

Numbers  of  computer  programs  are  of  keen  interest  but  again 
are  not  reflective  of  the  scope  of  total  support  requirements.  As  an 
example,  one  program  could  demand  several  hours  and  several  per¬ 
sonnel  to  execute  the  program  while  dozens  of  other  programs  could 
quickly  be  executed  by  a  single  person. 

1.4  TYPICAL  ATE  SYSTEM  FUNCTIONS 

This  section  is  addressed  in  twc  main  parts,  digital  testing  and 
engine  testing.  Coverage  of  these  tv  n  types  of  automatic  testing  repre¬ 
sents  the  bulk  of  software  support  problems  encountered  with  ATE.  Both 
types  of  testing  have  some  degree  o',  unilarity;  however,  engine  testing 
exercises  mechanical  and  pneumatic  testing  of  equipment  as  opposed  to 
pure  electronics  testing. 

1.4.1  Typical  Digital  ATE  Functions 

A  generalized  block  diagram  of  an  Automatic  Test  System  (ATS) 
is  outlined  in  Figure  1-2.  Although  the  ATS  can  vary  from  a  simple 
system  to  a  complex  system  either  the  simplified  or  the  complex  ATS 
accomplishes  tests  via  functions  as  outlined  in  the  figure. 

The  test  program  tape  or  disc  contains  the  coded  sequence  that 
activates  the  ATE  with  a  set  of  instructions  to  automatically  ascertain 
the  operational  readiness  condition  of  the  UUT,  and  if  faulty,  to  isolate 
the  fault  to  the  required  level  for  maintenance  action. 

Mechanical  and  electrical  connection  and  conditioning  between  the 
ATE  and  UUT  is  provided  by  the  Interface  Test  Adapter  (ITA).  The  Test 
Program  Instructions  (TPI)  provide  sequential  directives  for  execution 
of  a  test  program  to  include  set-up/tear-down  steps,  loading,  adjust¬ 
ments,  etc. 


In  summary,  the  UUT  is  exercised  by  the  ATE  through  the  ITA 
connection  by  the  test  program  as  amplified  by  the  test  program  instructions. 

In  normal  operational  methods,  the  test  requirements  and  UUT 
source  data  specified  in  the  Test  Requirements  Document  (TRD)  are  ana- 
lized  to  develop  a  test  strategy.  Based  on  the  test  strategy  the  test  design 
engineer  develops  the  detailed  test  logic  which  is  converted  to  the  test 
program  language.  This  UUT  program  is  processed  into  an  executable 
code  as  required  by  the  ATE  processor.  At  run  time,  the  test  program 
is  loaded  into  the  ATE's  Central  Processing  Unit  (CPU)  and  executed 
similarly  to  any  other  computer  program.  During  program  execution  one 
or  more  stimuli  are  selected  and  input  to  the  UUT.  The  UUT  response  is 
detected  and  measured  for  immediate  comparison  to  pre-established  pass/ 
fail  conditions  or,  in  a  few  cases,  the  response  is  recorded  for  future 
evaluation.  Results  of  all  the  comparisons  and  evaluations  indicate  the 
condition  of  the  UUT  and  either  passes  or  fails  the  UUT.  Test  data  and 
operator  instructions  may  be  Cathode  Ray  Tube  (CRT)  displayed  during 
program  execution  and  certain  operations  may  require  operator  actions 
to  be  entered  through  the  operator  interface.  When  testing  is  complete  the 
operator  is  notified  via  the  CRT  and  the  UUT  is  ready  for  removal  from 
the  test  system. 

The  previous  discussion  presented  a  simplified  description  of  how 

ATS's  function,  however  the  magnitude  of  testing  or  number  of  tests 

required  are  of  utmost  concern.  In  the  case  of  a  purely  digital  UUT  the 

total  number  of  different  test  combinations  possible  is  2*1  where  n  is 

the  number  of  inputs.  That  is,  a  UUT  with  10  inputs  would  represent 
10 

a  total  of  (2  =  1024)  different  test  combinations  possible.  Sequential 

logic  circuitry  would  represent  even  more  possible  combinations. 

Larger  numbers  of  inputs  (some  UUT's  contain  over  200  inputs)  are 
common  in  UUT's  and  the  magnitude  of  total  test  combinations  is 
astronomical  as  the  number  of  inputs  grow.  Additionally,  each  input 
combination  used  must  be  measured  for  UUT  response.  As  a  con¬ 
sequence,  the  test  program  must  be  carefully  constructed  and  executable 
in  rapid  sequence.  In  most  cases  the  central  processor  executes  thou¬ 
sands  of  test  combinations  within  a  few  minutes;  however,  seldom  is  a 
UUT  exhaustively  tested  (every  possible  combination  checked). 
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When  literally  thousands  of  tests  are  required,  it  is  illogical  and 
sometimes  impossible  to  use  unique,  separate  wires  for  each  test. 

Instead  a  switching  action  occurs  within  the  input/output  circuitry  function 
box.  Because  many  of  the  newer  UUT's  require  high  speed  checking,  the 
switching  frequency  can  approach  several  MHz  (million  cycles  per  second) 
and  thus  becomes  quite  complex.  At  high  frequency,  physical  separation 
becomes  a  problem  due  to  signal  attenuation.  Due  to  this,  many  of  the 
testers  are  moving  toward  circular  or  cylindrical  configurations  to 
physically  allow  each  test  to  use  equivalent  electrical  signals. 

1.4. 1.1  ATE  Complexity 

To  this  point  the  ATE  method  of  testing  seems  rather  simple  and 
straightforward;  however,  it  is  more  complex  than  it  appears.  The  next 
few  paragraphs  are  written  to  create  a  feeling  for  the  true  complexities 
involved. 


Figure  1-3  indicates  that  two  kinds  of  ATE  exist,  on-line  and  off¬ 
line.  On-line  is  defined  as  being  resident  in  the  UUT;  this  is  an  attempt 
at  prompting  the  UUT  to  check  itself.  The  functions  of  on-line  ATE  are 
as  indicated  in  the  figure. 

Off-line  ATE  is  depicted  in  Figure  1-3  aB  having  two  functions. 

An  alternate  way  of  thinking  of  off-line  functions  is:  control,  stimulus, 
measurement,  switching,  and  input/output.  Figure  1-4  gives  an 
indication  of  the  interfaces  for  these  functions  and  the  types  of  data 
exchanged  between  them. 

Examples  which  typically  apply  to  some  of  the  functional  blocks 
in  Figure  1-4  are; 

•  Stimulus  subsystem  -  pulse  generators,  waveform 
generators,  special  functions,  digital  signals. 

•  Measurement  subsystem  -  digital  multimeters,  counters/ 
timers,  waveform  analyzers,  spectrum  analyzers, 
distortion  analyzers,  transfer  function  analyzers, 
frequency  meters,  power  meters,  programmable 
oscilloscopes. 

s  Interface  test  adapters  -  can  vary  in  complexity  from 
simple  wiring  to  computer  controlled  parameters  for 
supplementing  short  falls  of  the  ATE. 


Figure  1-3.  Types  and  Functions  of  ATE 


Figure  1-4.  Off-line  Automatic  Test  System 


Testing  complexity  can  be  appreciated  when  the  following  situations 
are  considered. 

•  The  UUT  may  have  as  many  as  200  to  300  input  pins. 

•  Any  pin  may  have  at  least  one  stimulus  and  perhaps  more 
with  corresponding  measurements  of  the  UUT  reactions 
to  the  stimulus 

•  Switching  might  occur  at  variable  rates  up  to  50  MHz 
or  greater. 

•  Multiple  ITA's  could  apply  to  the  same  UUT  or  ATE. 

•  ITA  sophistication  may  require  its  own  separate  processor. 

•  The  type  of  testing  could  be  analog,  digital,  or  hybrid 
(a  mixture  of  analog  and  digital). 

An  additional  consideration  of  the  typical  ATE  functions  is  the 
testing  level  at  which  the  ATS  resides:  organizational,  intermediate, 
or  depot  level.  Typically  an  organizational  level  ATS  will  test  at  an 
avionics  subsystem  level  with  well-defined,  inflexible  test  programs. 

The  test  objective  at  this  level  is  to  determine  if  the  avionics  system 
functions  and,  if  not,  which  LRU  is  defective.  At  the  intermediate 
level  the  testing  is  accomplished  on  each  LRU  with  the  objective  of 
verifying  individual  LRU's  or  isolating  fault  to  the  SRU  level.  Finally, 
at  the  depot  the  test  objective  is  focused  on  verifying  the  SRU  and  to 
fault  isolate  at  a  lower  component  level.  As  the  ATS  are  applied  from 
organizational  level  up  to  depot  the  ATE  complexity,  flexibility,  and 
capability  increases.  Thus  the  simpler  ATE  tester  is  dispersed  to 
organizational  users  and  the  more  sophisticated  ATE  tester  resides 
at  the  depot. 

1.4. 1.2  Standard  ATE  Language 

In  the  late  1960's  Aeronautical  Radio  Inc.  (ARINC)  developed  a 
test  language  for  commercial  airline  use.  This  language  is  identified 
a 8  ATLAS  which  is  an  acronym  for  Abbreviated  Text  Language  for  All 
Systems.  Since  its  inception,  ATLAS  has  undergone  continuous  improve¬ 
ment  and  development  until  it  is  now  accepted  by  all  United  States 
services,  NATO,  DOD,  and  IEEE  as  the  standard  test  language. 
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The  principal  management  advantages  of  a  standard  Higher  Order 
Language  (HOL)  such  as  ATLAS  are; 

•  Easily  readable,  simplifies  program  documentation 
and  program  maintenance 

•  Reduced  training  requirements 

•  Reduced  programming  costs  (corporate  memory) 

•  Improved  communications  between  development/ 
manufacturing /test/and  maintenance  disciplines 

•  Reduces  non-recurring  costs  related  to  language 
development  and  maintenance 

ATLAS  is  a  UUT  oriented  language  as  opposed  to  other  languages 
that  are  "test  system"  oriented.  This  means  that  ATLAS  is  test  system 
independent  rather  than  tied  to  particular  testers  as  a  "test  system" 
language  would  be.  The  basic  elements  of  ATLAS  are: 

•  It  uses  fundamental  rules  of  syntax  and  semantics 
which  integrate  the  elements  into  a  language 

•  Input/output  features  the  capability  for  data  to  be 
communicated  between  the  test  system  and  the 
system  operator 

•  It  features  data  processing  capability  such  aB  calculations, 
data  storage,  program  structure,  and  program  control 

s  It  uses  a  precise  set  of  actions,  taken  relative  to  a  UUT, 
such  as  applying  a  stimulus,  connecting  a  signal,  or 
making  a  measurement 

•  It  uses  a  precise  set  of  electrical  mechanical  signals 
which  are  either  applied  to  the  UUT  or  measured 
from  the  UUT. 

The  last  two  basic  elements  primarily  distinguish  ATLAS  from 
other  test  languages.  ATLAS  is  not  intended  as  a  language  for  a  par¬ 
ticular  tester,  or  set  of  testers.  Rather,  ATLAS  has  a  primary  purpose 
of  specifying  tests  clearly  and  unambiguously. 

There  is  some  concern  that  ATLAS  is  difficult  to  compile  and  that 
a  large  number  of  different  ATLAS  versions  exist.  ATLAS  is  a  large 
language  which  can  be  applied  to  analog,  digital,  or  hybrid  testing  with 
varying  degrees  of  sophistication.  The  result  is  that  individual  users 
adopt  and  compile  a  subset  of  ATLAS  which  applies  to  their  needs  only. 
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Thus  many  different  "configurations"  exist  but  all  use  the  standard 
language.  For  a  period  of  several  years  the  ATLAS  compilers  were 
error  prone  and  slow;  however,  within  the  past  few  years  the  ATLAS 
compilers  have  become  stable  and  the  compilations  are  no  longer  as 
difficult. 

The  adoption  of  ATLAS  as  a  standard  test  language  has  been  com¬ 
pared  to  the  adoption  of  English  as  the  worldwide  standard  for  inter¬ 
national  airport  traffic  controllers.  Neither  adoption  is  to  say  that  it 
is  the  optimum  choice  of  language  but,  rather,  it  is  the  accepted  standard. 

1.4.2  Typical  ATE  System  Functions  in  Engine  Testing 

Closed  loop,  computer -controlled  testing  of  overhauled  jet  aircraft 
engines  is  a  growing  technology  within  the  Air  Force.  Since  the  mid- 
1960's,  when  automated  engine  testing  was  just  beginning,  the  Air  Force 
has  continuously  pursued  the  technique  within  its  organization  and 
encouraged  computer  and  test  equipment  manufacturers  to  do  the  same. 
State-of-the-art  technology  in  computer-controlled  engine  testing  is 
probably  most  visible  at  SA-ALC  and  OC-ALC.  Both  ALC's  are  exten¬ 
sively  involved  in  the  testing  of  overhauled  engines. 

All  aspects  of  engine  test,  from  preliminary  checkout  through 
various  manufacturer- specified  routines,  are  under  computer  control 
with  a  system  known  as  Pacer  Comet.  The  system  was  initiated  in  1969 
and  since  then  has  undergone  near  continuous  improvements.  Pacer 
Comet  is  used,  for  example,  to  test  more  than  1,000  engines  each  year 
at  SA-ALC. 

Originally  each  engine  type  required  different  software  programs, 
an  average  of  23,  in  order  to  test  and  measure  particular  performance 
and  engine  operation.  Existing  test  cell  equipment  and  wiring  had  to 
be  modified  or  replaced  and  all  subsystems  of  the  process  integrated. 

As  personnel  progressed  through  the  various  phases  of  system 
development,  they  constantly  identified  new  techniques,  software  enhance¬ 
ments,  and  equipment  interfaces  that  would  enhance  engine  throttle  control, 
increase  test  data  reliability,  and  make  additional  use  of  the  information. 
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Now  there  is  a  second  generation  system,  Pacer  Comet  Mark  II,  that 
enables  one  computer  to  handle  four  test  cells  simultaneously  while 
providing  additional  operating  data.  Figure  1-5  shows  a  block  diagram 
of  the  Pacer  Comet  Mark  II  test  system. 

Pacer  Comet  provides  dramatic  results  when  compared  with  the 
manual  operation.  For  example,  during  the  manual -conducted  test 
of  a  J-79  engine,  the  operator  would  move  the  throttle  each  time  a 
change  in  engine  speed  was  required  while  carefully  adhering  to 
manufacturer's  specifications  and  the  technical  order  of  the  test. 
Furthermore,  the  operator  would  make  approximately  700  instrument 
readings  and  recordings,  25  chart  plots,  150  calculations,  and  40  entries 
to  the  engine  log  sheet.  For  more  complex  engines  these  taBks  could 
increase  three-fold  to  the  point  that  even  the  most  experienced  operator 
was  inundated  with  the  data.  About  40  man-hours  of  post-test  data 
calculations  were  required.  Pacer  Comet  enabled  these  same  calculations 
and  more  to  be  available  within  a  few  seconds. 

Up  to  four  different  engine-type  tests  can  be  handled  simultaneously 
and  asynchronously  by  a  single  host  computer.  Communications  between 
the  host  and  cell  systems  are  supported  by  a  data  link,  and  both  systems 
support  CRT's  as  the  primary  interface  between  operator  and  test  cell, 
and  the  operator  and  host  computer. 

In  a  typical  test,  after  the  operator  has  entered  identification  data 
the  computer  takes  over.  It  checks  all  contact  points  for  proper  open 
or  closed  status,  performs  a  "confidence  check"  of  all  wiring  and  con¬ 
nections,  and  initiates  a  "dry  start"  that  pushes  air  through  the  engine 
to  check  all  systems.  Then  it  starts  the  engine  and  a  series  of  prelimi¬ 
nary  tests  are  made. 

The  operator  "trims"  the  engine  for  idle  speed  and  the  computer 
checks  oil  pressure,  exhaust  gas  temperature,  vibration,  speed,  and 
bearing -vent  pressure  and  displays  the  results  to  the  operator.  In  the 


Figure  1-5.  Block  Diagram  of  Pacer  Comet 
Mark  II  Test  System 
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next  test  phase,  the  computer  checks  bleed-valve  operation  and  returns 
the  engine  to  idle.  The  computer  then  activates  maximum  throttle  posi¬ 
tion  and  the  anti-ice  system  is  checked  for  possible  malfunction. 

Next,  the  engine  is  run  by  the  computer  through  a  series  of  accel¬ 
eration  and  deceleration  phases  in  both  the  "primary"  and  "emergency" 
fuel  control  systems,  with  timed  runs  made  in  between  for  stabilization 
purposes.  To  complete  the  preliminary  test,  the  engine  is  shut  down 
and  mechanics  enter  the  test  cell  to  check  for  oil  and  fuel  leaks,  con¬ 
tamination  of  strainers  and  filters,  or  similar  faults. 

In  the  performance  runs  that  follow,  more  comprehensive  checks 
are  made  according  to  a  manufacturer-prescribed  series  of  tests. 
Operating  characteristics  such  as  specific  fuel  consumption  resulting 
from  different  thrusts  and  rotor  speeds  and  internal  pressure  ratios 
are  measured,  calculated,  and  recorded.  Normally,  three  power  runs 
are  made  between  75  percent  of  normal  rated  power  and  military  power. 
Each  of  these  runs  lasts  three  to  five  minutes  and  all  data  is  computer 
corrected  to  standard  day  sea  level  conditions  for  proper  comparison. 

To  complete  the  test,  the  final  shakedown  consists  of  another 
series  of  timed  runs  interspersed  with  acceleration,  deceleration, 
vibration,  and  exhaust  gas  temperature  checks,  followed  by  another 
physical  inspection  after  shutdown. 

The  system  also  provides  a  complete  recorded  history  of  the  test 
performed.  This  includes  documentation  to  support  acceptance  or  rejec¬ 
tion  of  the  engine.  As  test  data  is  accumulated.  Pacer  Comet  provides 
a  diagnostic  evaluation  of  the  rejected  engine  to  determine  rework 
requirements  and  provide  for  analysis  to  predict  engine  trends. 

When  an  engine  does  not  meet  all  performance  requirements,  it 
goes  to  the  "penalty  line"  where  mechanics  and  machinists  make  the 
necessary  repairs  and  adjustments.  Then  it  is  completely  retested. 
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An  additional  mode  of  testing  can  be  thought  of  as  a  subset  of 
engine  testing  because  it  addresses  the  automatic  testing  of  separate 
engine  components.  The  F100  engine  which  is  used  by  the  F-15  and  F-l6 
aircraft  is  component  and  "whole"  engine  tested.  This  engine  type  is 
used  here  as  an  example  of  engine  component  testing  and  the  kinds  of 
activities  involved. 

The  F100  engine  requires  a  family  of  depot  level  test  stands  to 
support  maintenance  testing,  upkeep,  and  modification  implementation  on 
the  engine  modules  and  accessories.  Associated  ATE  was  selected  in 
conformance  with  the  basic  maintenance  concept  which  requires  rapid 
assessment  and  identification  of  failure  to  a  single  item  or  component 
level.  Fault  corrections  are  to  the  piece  part  level. 

The  unified  control,  the  engine  electronic  control,  the  augmentor 
spray  manifolds,  and  the  air  cooled  turbine  components  comprise  the 
F100  engine  modules  and  accessories  requiring  computer-aided  test 
stands.  Each  of  these  is  briefly  addressed  in  the  following  paragraphs. 

The  F100  engine  unified  control  test  stand  system  is  used  to  per¬ 
form  overhaul  performance  testing  of  the  complete  unified  control,  or 
the  main  section  and  augmentor  section  separately,  or  in  any  combi¬ 
nation  of  the  three  as  UUT's.  These  test  stands  are  operated  through 
a  Data  General  NOVA  820  computer  using  both  its  own  executive  and 
self-test  software.  The  computer  has  four  ports  which  provide  a 
physical/electrical  interface  through  which  any  combination  of  the  test 
stands  with  appropriate  cables,  specified  port  configured  computer  pro¬ 
gram,  and  interface  hardware  may  be  configured.  The  physical  con¬ 
figuration  can  range  from  one  operating  test  stand  with  three  empty  ports 
to  four  operating  test  stands  or  any  intermediary  combination. 

The  F100  engine  augmentor  spray  manifold  test  stand  system  is 
used  to  verify  the  performance  characteristics  of  the  engine's  five 
augmentor  spray  manifold  assemblies.  The  system  uses  a  Data  General 
NOVA  820  central  computer  which  can  operate  up  to  eight  test  stands. 
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Software  programs  remain  unchanged  independent  of  the  number  of 
test  stands  operated;  however,  additional  hardware  interface  adapters 
and  controllers  are  required.  A  manifold  configuration  change  may 
require  software  changes;  therefore,  correlation  between  the  two  is 
maintained  via  configuration  management  procedures. 

The  air  cooled  turbine  component  test  stand  verifies  the  per¬ 
formance  characteristics  of  the  first  and  second  stage  turbine  rotor 
blades,  and  the  first  and  second  stage  turbine  stator  vanes.  The  com¬ 
puter  resources  existing  within  this  test  stand  are  an  embedded  FX 
Systems  Mark  I  calculator  and  a  loadable  paper  type.  The  test  stand 
is  controlled  by  the  FX  Systems  Mark  I  calculator.  Although  originally 
lacking  a  diagnostic  capability  for  the  controller,  a  procurement  was 
made  to  enable  troubleshooting  and  maintenance  of  the  controller  and 
to  acquire  several  software  capabilities. 

Only  two  of  the  engine  electronic  control  test  stands  utilize  com¬ 
puter  resources.  These  test  stands  accomplish  test,  troubleshoot, 
fault  isolate,  and  simulate  operational  stimuli.  Different  test  stand 
configurations  exist  and  each  has  a  distinctly  different  capability  to 
fault  isolate. 

1.5  ATE  SYSTEM  RELATIONSHIP  TO  ROLES /MISSIONS 

The  importance  of  testing  is  equally  as  important  to  the  Air  Force 
as  the  importance  of  any  particular  unit  under  test  because  it  helps  the 
maintenance  keep  the  UUT  operative.  Testing,  whether  automatic  or 
manual,  serves  to  verify  the  UUT  is  capable  of  performing  the  task(s) 
for  which  it  is  intended.  UUT's  generally  are  becoming  increasingly 
complex  and  thousands  of  tests  must  be  routinely  performed  on  them  to 
verify  their  operational  state.  Because  manual  testing  techniques  are 
often  impractical  due  to  the  nature  and  number  of  tests  required,  some 
form  of  automatic  testing  is  necessary. 
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It  has  been  estimated  that  automatic  test  systems  account  for 

approximately  75  percent  of  the  total  support  equipment  acquisition  costs 

for  the  Air  Force^  \  As  an  example  of  the  magnitude  of  ATE  costs,  the 

ATE  development  and  acquisition  costs  for  the  F-16  are  estimated  at 
/ 2 ) 

$600  million.  '  Total  ATE  inventory  costs  for  AFLC  are  currently 

estimated  at  more  than  $10  billion.  Annual  costs  for  Air  Force 

support  equipment  have  exceeded  $1  billion  since  1975  and  promise  to 

(41 

soar  higher  (now  projected  at  $3  billion).'  ’  The  cost  of  support  equip¬ 
ment,  including  ATE,  is  highly  significant  and  warrants  constant 
planning  and  management  attention. 

Today's  weapon  systems,  including  their  avionics  subsystems, 
are  complex  and  their  capabilities  are  crucial  to  the  success  of  national 
offensive  and  defensive  strategies.  The  F-15  is  an  example  of  this 
complexity  and  Table  1-2  illustrates  the  number  of  Interface  Test 
Adapters  (ITA's),  test  programs,  and  UUT's  which  are  part  of  the 
correspondingly  complex  ATE  for  this  weapon  system. 


^  AUTOTESTCON  '79  presentation  by  Capt.  Floyd  D.  Long,  Jr., 
Support  Equipment  SPO,  ASD,WPAFB. 

(21 

An  Operating  and  Support  Cost  Model  for  Avionics  Automatic  Test 
Equipment,  Masters  Thesis  by  J.A.  Guerra,  A.  J.  Lesko  and 
J.G.  Pereira. 

(31 

Informal  discussions  at  San  Antonio  ALC,  May  1980. 

^  Industry /Joint  Services  Automatic  Test  Project  Executive 
Summary,  December  1979. 
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Table  1-2.  F-15  ATE  ITA/TP/UUT  Overview1 


ATE  Station 

ITA's 

Test 

Programs 

UUT's/WUC 

HIADTS  Digital 

53 

217 

130 

Analog 

61 

322 

154 

IF/VM 

24 

64 

22 

HPADTS 

TBD 

TBD 

22 

Multifunction 

17 

63 

51 

COMETS  Digital 

8 

24 

31 

INS  (Litton) 

31 

32 

32 

AAI  5500 

5 

7 

7 

TEWS  Depot 

TBD 

TBD 

100 

AIS  Computer 

28 

45 

29 

AIS  Displays 

12 

39 

12 

AIS  Microwave 

5 

19 

5 

TITE 

TBD 

24 

24 

AIS  AGE 

Kelly  Microwave 

100 

123 

123 

Power  Supply 

56 

97 

97 

GEN  RAD  1792D/1096 

42 

165 

165 

COMETS  AGE 

TBD 

359 

377 

*  Reference  data  only;  subject  to  change  throughout  life  cycle  of  F-15. 


Generically,  automatic  testing  serves  to: 

•  Verify  the  components  of  weapon  systems 

•  Provide  fault  isolation  of  component  discrepancies 

•  Minimize  spare  levels  by  quickly  indicating  faults 
and  expediting  repair  activities 

•  Reduce  the  demand  for  highly  trained  technicians 

Automatic  testing  is  growing  in  costs  and  importance  to  the  Air 
Force-  Increased  attention  must  be  paid  to  existing  policies,  procedures, 
planning,  and  management  to  ensure  that  weapon  systems  operationally 
function  as  required  and  that  ATE  is  not  in  any  way  a  deterrent  to 
weapon  system  availability. 


i 
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2.  ATE  SUPPORT  REQUIREMENTS 


Automatic  test  equipment  provides  support  to  a  wide  variety  of 
Air  Force  weapon  systems  and  to  their  components.  The  automatic 
testing  can  be  of  a  nature  to  check  software  and/or  hardware  of  the 
weapon  systems  or  even  to  check  the  ATE  itself.  As  mentioned  earlier 
in  this  document  there  are  two  types  of  ATE  testing,  on-line  and  off-line. 
Either  type  requires  support;  this  section  addresses  support  requirements 
as  they  generally  apply  to  both  types.  The  generic  requirements  of 
change,  change  analysis  and  specification,  engineering  development  and 
unit  test,  system  integration  and  test,  change  documentation,  and  cer¬ 
tification  and  distribution  apply  to  ATE  as  well  as  to  all  other  ECS 
categories.  Table  2-1  summarizes  the  remarks  on  each  support  require¬ 
ment. 

2.1  ECS  CHANGE 

2.1.1  Receive  and  Process  Requests 

Changes  to  ATE  or  ATE  software  can  be  of  a  hardware  component, 
test  program,  control  program,  or  support  program  nature.  Changes 
to  ATE  hardware  occur  because  of  technology  improvement  or  to  over¬ 
come  a  test  limitation.  Software  changes  most  commonly  occur  to  the 
UUT  test  programs  because  of  their  lack  of  testing  depth  or  inaccurate 
testing;  however,  changes  can  occur  to  the  ATE's  control  and  support 
software  programs.  Discovery  of  the  need  for  change  may  be  made  at 
the  depot,  intermediate,  or  organizational  testing  levels.  Change  noti¬ 
fication  is  usually  made  via  a  Material  Deficiency  Report  (MDR)  sub¬ 
mitted  to  the  UUT  item  manager  or  to  the  ATE  system  manager;  however, 
a  change  notification  may  also  result  from  an  Engineering  Change 
Proposal  (ECP)  to  modify  the  weapon  system  design. 
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Table  2-1.  ATE  System  Support  Requirements 
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Preliminary  Design  •  Alternative  designs  are  considered  and  tne  most 

promising  one  is  selected. 

Detailed  Design  •  This  activity  establishes  the  means  of  developmen 

and  technical  verification. 

Generate  Change  Proposal  •  This  step  formalises  its  work  package  for  organic 


Table  2-1.  ATE  System  Support  Requirements  (Concluded) 


1 
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2.1.2  Preliminary  Analysis  and  Problem/Deficiency  Definition 

Upon  receipt  of  the  ATE  change  request,  the  responsible  agency 
initiates  change  control  procedures  and  an  engineering  analysis  of  the 
change.  Preliminary  analysis  of  any  impact  to  existing  capabilities  ;s 
accomplished.  If  no  abortive  type  impacts  are  recognized,  the  tech"' cal 
change  to  the  ATE  is  defined. 

2.1.3  Preliminary  Resource  Allocation  and  Scheduling 

A  relative  priority  of  the  ATE  change  versus  other  change  requests 
is  established.  For  all  accepted  changes  the  responsible  agency  ensures 
that  resources  involving  organic  and/or  contractor  sources  are  allocated 
and  a  preliminary  schedule  for  completion  is  published. 

2.2  CHANGE  ANALYSIS  AND  SPECIFICATION 

2.2.1  Establish  Change  Feasibility 

Feasibility  is  determined  by  assessing  if  the  impacts  to  the  exist¬ 
ing  system  are  worth  the  penalties  to  incorporate  the  change.  In  most 
ATE  cases  the  feasibility  is  established  by  this  time  since  the  change 
may  only  necessitate  an  additional  test  sequence  or  a  new  set  of  input 
stimuli. 

2.2.2  Requirements  Decomposition/Definition 

For  simple  changes  this  step  would  have  been  completed  by 
satisfying  the  requirement  discussed  in  paragraph  2.1.2.  But  for  hard¬ 
ware  or  control /support  software  changes  it  may  be  necessary  to  break 
the  development  task  into  discrete  work  packages  and  a  well-defined 
design  approach. 

2.2.3  Preliminary  Design 

For  major  changes,  alternative  designs  may  be  conceived  and 
analyzed  for  selection  of  the  most  promising  approach. 

2.2.4  Detailed  Design 

This  activity  selects  a  design  approach,  establishes  the  means  of 
development  and  technical  verification,  and  identifies  problems  and 
procedures. 
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2.2.5  Generate  Change  Proposal 


When  the  task  solution  is  ready  for  formal  approval,  a  work  package 
is  developed  and  locally  approved. 

2.3  ENGINEERING  DEVELOPMENT  AND  UNIT  TEST 

2.3.1  Develop  the  Change 

* 

The  change  proposal  is  accomplished  through  use  of  separate  or 
composite  in-house  and  contractor  resources.  These  resources  are 
spread  across  the  spectrum  of  work  packages  and  normal  development 
practices  are  applied. 

2.3.2  Perform  Engineering  Tests 

All  work  units  of  hardware  and/or  software  are  separately  tested 
during  development  to  verify  the  desired  technical  functions  of  the 
changes  are  working,  the  engineering  integrity  of  the  programs  are 
valid,  and  no  unusual  or  problem  areas  persist  with  the  change. 

2.4  SYSTEM  INTEGRATION  AND  TEST 

2.4.1  Test  ATE  System  Performance 

Testing  of  the  ATE  after  incorporation  of  the  change  is  necessary 
to  get  a  measure  of  confidence  the  revised  ATE  system  performs  as 
desired  and  without  impacting  previously  established  capability. 

2.4.2  Test  Weapon  System  Performance 

This  specific  requirement  does  not  apply  to  ATE. 

2.4.3  Produce  Test  Results 

Subsequent  to  test  completion,  sufficient  data  for  analysis  and 
test  results  are  compiled  and  published  in  written  format. 

2.5  CHANGE  DOCUMENTATION 
2.5.1  Document  ATE  Changes 

All  working  documents  must  be  updated  to  provide  a  valid  working 
baseline  fo  reference  material  for  further  support. 
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Update  all  baseline  documentation  to  include  hardware  drawings, 
associated  software  descriptions,  and  any  pertinent  procedural  changes. 


2.5.3  Configuration  Control  * 

Throughout  the  final  stages  of  developing  and  testing,  the  various 
trial  and  final  versions  must  be  rigidly  controlled  so  that  Incremental 
progress  is  benchmarked  and  catastrophic  failure  will  not  occur. 

2.6  CERTIFICATION  AND  DISTRIBUTION 

2.6.1  Certify  Documentation 

This  is  an  administrative  function  to  recognize  the  revised  ATE 
and  its  baseline  description. 

2.6.2  Distribute  Revised  ATE  System  Data 

This  is  a  requirement  for  multi-user  ATE  for  which  the  ATE 
applies  commonly  to  different  avionic  systems. 

2.6.3  Provide  Installation  Procedures/Instructions 

This  may  be  a  requirement  if  the  hardware  change  is  significantly 
complex  or  if  the  system  change  necessitates  new  operating  procedures. 


3.  ATE  SUPPORT  CONCEPT  DESCRIPTION 


Support  for  ATE  systems  is  generally  organically  accomplished 
by  AFLC  using  all  five  Air  Logistics  Centers  and  the  Aerospace 
Guidance  and  Metrology  Center  (AGMC)  to  provide  various  aspects  of 
the  support.  Certain  ATE  support  may  require  contractor  resources 
whose  overall  activities  are  AFLC -managed  after  PMRT,  but  typically 
are  managed  by  AFSC  prior  to  PMRT.  Discussions  in  this  section  will 
address  the  logistics  management,  organizational,  and  system  change 
portions  of  the  ATE  support  concept. 

3.1  MANAGEMENT  CONCEPT 

AFLCR  523-1  establishes  policy  governing  mission  assignments 
to  all  AFLC  activities.  Using  general  criteria  such  as  existing  invest¬ 
ment  in  skills  and  facilities,  technical/management  skills,  workload 
balance,  etc.,  each  ALC  is  assigned  specific  management  tasks. 
Management  of  ATE  is  assigned  in  accordance  with  AFLCR  523-1  and 
new  ATE  entering  the  inventory  is  normally  assigned  to  the  ATE  SM. 
The  associated  svipport  software  is  assigned  with  the  ATE  management 
assignment  and  supported  in  accordance  with  AFLCR  66-27.  ATE 
control  software  is  managed  with  the  tester.  The  ATE  equipment  and 
its  support  and  control  software  are  generally  assigned  to  San  Antonio 
ALC  for  management.  This  assignment  is  depicted  in  Figure  3-1  as 
the  portion  of  the  automatic  test  system  that  is  outlined  with  the  heavy 
solid  lines  and  labelled  automatic  test  equipment. 

ATE  test  software  (UUT  test  program)  is  managed  by  the  UUT 
manager  who  may  be  located  at  any  ALC  and  who  manages  the  software 
in  accordance  with  the  Computer  Program  Configuration  Item  (CPCI) 


procedures  outlined  by  AFR  800-14.  Rapid  deficiency  resolution  may 
necessitate  a  deviation  from  the  CPCI  control  route.  All  UUT  soft¬ 
ware  changes/implementations  are  required  to  be  coordinated  with 
the  assigned  ATE  manager  if  the  hardware  is  impacted. 

Interface  Test  Adapters  (ITA)  are  also  managed  by  the  UUT 
manager  unless:  (a)  the  equipment  is  purchased  to  provide  a  general 
purpose  capability  for  the  tester  or  (b)  an  ITA  (not  excepted  by  (a)  ) 
is  used  in  testing  UUT's  assigned  to  two  or  more  IM/SM  divisions. 

The  UUT  manager  responsibility  items  are  indicated  by  the  heavily 
lined  boxes  in  Figure  3-2. 

Many  of  the  computers  and  peripheral  equipment  used  within 
ATE  are  federally  stock  classed  as  FSG-70  which  is  the  management 
responsibility  of  WR-ALC  Item  Management  Division  (MMI).  In 
many  cases  this  responsibility  includes  some  or  all  of  the  support  and 
control  software.  The  heavily  lined  box  shown  in  Figure  3-3  indicates 
this  assignment  responsibility  and  the  FSG-70  manager's  role  in  this 
regard  is  not  generally  recognized. 

AGMC  is  assigned  the  responsibility  to  determine  requirements 
for  and  accomplish  equipment  calibration  support. 

Tables  3-1  and  3-2  use  the  F-15  situation  as  an  example  to 
illustrate  the  application  of  the  ATE  management  concept.  These 
tables  should  not  be  construed  as  entirely  describing  the  F-15  manage¬ 
ment  assignment  situation,  but  only  a  portion  of  that  assignment. 

Table  3-1  shows  the  AFLC  location  of  each  ATE  station  and  the 
approximate  quantities  of  UUT's  tested  on  each  station  managed  by 
each  ALC. 


Table  3-2  shows  the  distribution  of  ATE  stations  by  ALC  as  results 
from  technical  repair  center  assignment.  Associated  with  each  station 
is  a  set  of  control  and  self-test  software.  Management  of  this  software 
is  the  responsibility  of  the  ATE  SM  typically  located  at  SA-ALC. 

3.1.1  Management  Concept  Assessment 

Management  relationships  of  the  ATE  responsible  agencies  are  closely 
interrelated  and  complex.  The  examples  given  in  this  section  have 
illustrated  the  complexities  associated  with  the  F-15  weapon  system  only. 
When  all  other  weapon  systems  are  overlayed  together,  the  complexity 
of  any  particular  ALC  or  of  them  all  is  intricate.  Exceptions  also  com¬ 
pound  the  complexity  when  certain  weapon  systems  necessitate  separation 
of  their  unique  ATE  from  the  normal  management  assignment  mechanisms. 

The  current  ATE  management  concept  provides  ample  visibility  of 
items  and  systems  although  the  overall  management  problems  are  com¬ 
plex.  The  interrelationships  of  the  item  and  system  assignments  mask 
grass  root  causes  of  any  management  problems.  Thus  the  rectification  of 
problems  becomes  subtle  to  conceive  and  implement. 

3.2  LOGISTICS  SUPPORT  CONCEPT 

AFLC  exerts  influence  on  automatic  test  system  acquisition  to  ensure 
that  logistics  support  of  each  element  of  the  ATS  is  accounted  for,  from 
the  beginning  of  the  weapon  system  conceptual  phase  through  the  operational 
application.  The  concept  requires  the  following.  The  assigned  ATE  SM 
is  responsible  for  planning  the  transition  of  the  ATE  from  the  acquisition 
agency  to  AFLC;  the  UUT  managers  are  responsible  for  Test  Requirements 
Documents  (TRD),  back-up  source  data,  ITA's,  and  the  UUT  test  pro¬ 
grams  for  their  assigned  UUT's;  and  spare  and  repair  quantities  are  indi¬ 
cated  through  use  of  the  standard  automated  data  systems  used  by  AFLC. 

The  FSG-70  manager  (WR-ALC/MMI)  is  responsible  for  logistics  support 
of  the  ATS  general  purpose  computer  and  peripherals. 


3.3  ATE  SOFTWARE  CHANGE  PROCESS/CONCEPT 


The  ATE  change  process  is  quite  intricate  to  describe.  Consequently 
this  section  is  structured  to  provide  rationale  for  ATE  software  changes 
and  then  a  description  of  the  change  process  used. 

3.3.1  ATE  Software  Changes 

Changes  to  ATE  can  involve  digital,  analog,  or  hybrid  functional 
testing  accomplished  through  built-in-test,  self-test,  diagnostic  tests, 
fault  isolation,  ATE  control  software,  ATE  support  software,  or  the 
UUT  test  programs.  Categorically,  changes  to  ATE  occur  because  of 
a  few  basic  reasons:  the  ATE  incorrectly  tests,  the  ATE  doesn't  test 
to  the  extent  it  should  or  the  ATE  overtests  (consistently  indicates 
failures  when  none  exist).  Thus  it  is  appropriate  that  automatic  testing 
be  briefly  discussed  prior  to  discussing  ATE  changes. 

Generally,  automatic  testing  is  either  of  a  functional  or  a  diagnostic 
nature.  Functionally  testing  a  UUT  simply  determines  if  the  UUT  oper¬ 
ates  correctly.  This  is  accomplished  by  applying  a  variety  of  input 
stimuli  to  the  UUT  and  observing  or  measuring  the  UUT  responses  to 
the  stimuli.  As  the  test  program  steps  the  UUT  through  its  logic  states, 
the  responses  are  compared  against  correct  responses.  As  long  as  the 
responses  match  (measured  response  against  the  expected  or  known 
response)  the  UUT  is  passing,  [f  a  mismatch  occurs,  the  UUT  indicates 
a  failure  and  must  be  repaired.  Certain  UUT's  require  testing  at 
normal  UUT  operating  speed  which  means  the  input  stimuli  must  occur 
at  that  rate  (dynamic  testing).  Other  UUT's  can  be  meaningfully  tested 
at  a  relatively  slow  rate  compared  to  normal  UUT  operation  (static  testing) 
In  either  static  or  dynamic  testing  the  sole  purpose  of  functional  testing 
is  to  verify  the  UUT  operational  state. 

Assuming  a  functional  failure  of  the  UUT,  diagnostic  information 
must  be  provided  to  assist  the  repair  agency  in  determining  the  deficient 
area  (fault  isolation)  of  the  UUT  so  repairs  can  be  accomplished.  There 
are  two  techniques  that  are  commonly  used  in  fault  isolation,  fault 
dictionary  lookup  and  computer  guided  probe. 
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A  fault  dictionary  contains  hundreds  of  response  patterns  that  are 
a  result  of  intentionally  inserted  faults.  If  a  match  occurs  between  the 
test  situation  and  one  of  the  patterns,  then  this  indicates  a  failed  com¬ 
ponent  and  thus  isolates  the  fault.  It  is  possible  to  see  that  all  of  the 
possible  fault  patterns  (signatures)  for  a  UUT  may  not  be  contained  in 
the  dictionary.  It  may  be  necessary  to  introduce  more  fault  patterns 
to  adequately  check  the  UUT  (a  UUT  test  program  change).  The  advan¬ 
tage  of  fault  dictionary  lookup  diagnosis  is  that  no  operator  interaction 
is  necessary  during  the  test  and  the  process  is  fast.  A  disadvantage 
is  the  dictionary  is  inherently  incomplete  and  limited  in  fault  isolation 
by  the  circuit  topology  (complexity  and  layout).  Sometimes  the  results 
provide  inaccurate  or  redundant  fault  isolation  indications. 

A  computer  guided  probe  does  not  depend  on  stored  data  of  faulty 
UUT  responses.  It  uses  data  of  a  properly  functioning  UUT  and  com¬ 
pares  it  to  data  read  by  the  probe.  A  mismatch  means  the  probed  area 
is  faulty.  Diagnostic  accuracy  using  the  probe  is  much  better  than  the 
fault  dictionary  method,  but  the  probe  is  operator  dependent  and  there¬ 
fore  slow  and  prone  to  human  error.  Additionally,  the  existence  of 
numerous  feedback  loops  can  limit  the  computer  to  the  diagnosis  of 
many  modes,  rather  than  one,  and  intermittent  faults  are  not  always 
detected.  These  shortcomings  can  prompt  computer  program  changes. 

Perhaps  the  most  promising  approach  for  future  ATE  systems 
is  a  combination  of  fault  dictionary  and  an  interactive  computer  probe. 
Complementing  the  fault  dictionary,  probing  improves  problem  resolu¬ 
tion  when  the  dictionary  approach  cannot  isolate  to  a  single  mode. 

UUT's  may  typically  contain  from  20  to  200  input  pins  with  many, 
many  circuits  on  a  card.  Exhaustive  testing  is  impractical,  if  not 
impossible,  thus  a  means  of  generating  only  the  most  meaningful  input 
stimuli  is  required.  Automatic  Test  Pattern  Generators  (ATPG)  are 
used  as  a  technique  whereby  the  ATPG  system  software  generates  UUT 
test  input  stimuli  based  upon  an  interpretation  of  the  logic  contained 
within  the  UUT.  No  ATPG  can  claim  to  generate  tests  for  every  possible 
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UUT  circuit,  and  sometimes  the  UUT  performs  in  a  manner  that  bypasses 
the  testing  extent  of  the  ATPG  stimuli.  Identification  of  such  a  test 
omission  necessitates  a  UUT  test  program  change.  ATPG  systems 
provide  the  test  programmer  with  varying  degrees  of  aid  depending  on 
the  UUT  circuit  nature  and  complexity.  Table  3-3  is  included  as  an 
example  of  2  number  of  ATPG's  that  can  apply  to  a  weapon  system  and 
its  associated  ATE.  In  this  case  the  weapon  system  is  the  F-15. 

As  can  be  seen  from  the  discussion  to  this  point,  many  factors  can 
cause  ATE  software  changes.  The  changes  occur  primarily  because  the 
test  programs  are  not  perfectly  developed.  When  software  changes  are 
required  the  algorithms  involved  are  not  generally  classified  as  tech¬ 
nically  complex,  although  certain  exceptions  exist  such  as  a  test  to 
exercise  a  complex  curve  fitting  algorithm.  The  difficulty  comes  pri¬ 
marily  from  isolating  the  single  fix  point  from  such  a  myriad  of 
possibilities. 

Hardware  changes  can  occur,  for  example,  when  a  measurement 
tool  is  not  fast  enov gh  or  the  interface  test  adapter  does  not  properly 
link  the  ATE  and  the  UUT.  In  many  cases  it  is  prevalent  and  easier 
to  make  changes  to  the  IT  A  in  lieu  of  the  ATE.  Such  actions  complicate 
the  ITA  and  in  certain  instances  the  ITA  demands  an  internal  processor. 
Hardware  changes  are  of  direct  interest  to  this  study  only  when  they 
require  or  influence  ATE  software  changes. 

The  largest  single  cause  of  ATE  software  changes  occurs  becauss 
of  incomplete  UUT  test  programs.  Incompleteness  is  caused  primarily 
by  inadequate  Test  Requirements  Documents  (TRD's)  or  a  failure  of  the 
acquisition  agency  to  implement  adequate  TRD  s  with  the  weapon  system 
acquisition.  This  situation  is  so  rampant  that  approximately  25  to  40 
percent  of  the  UUT  test  programs  will  not  function  when  initially  received 
at  the  Technical  Repair  Center  (TRC)  at  WR-ALC  and  75  to  90  percent 
of  the  test  programs  require  major  modification  within  two  years  after 
TRC  receipt.  Additionally,  software  changes  are  often  required  because 
of  incomplete  development  by  the  weapon  system  contractor  or  the  test 
program  contractor.  The  inadequate  TRD  situation  is  further  addressed 
in  Section  5  of  this  rep^t. 
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Table  3-3.  Automatic  Test  Pattern  Generators 
For  F-15  ATE 


ATE  Family 

ATPG  Name  or  Owning 
Contractors 

No.  ofUUT's 

Honeywell  ADTS 

IBM  ATPG 

7 

Honeywell  (HI-FADS) 

5 

Digital 

Hughes  SATGEN 

46 

Texas  Instruments  TI-ATPG 

20 

COMETS  Digital  -  66FC 

LASAR  (Navy  Version) 

5 

COMETS  -  6861 

Bendix  ATPG/MCAIR 

56 

TEWS  Depot  Station 

Test  Aids  III 

45 

HP  ADTS 

T est  Aids  III 

14 
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AFLCR  66-37  lists  five  principal  activities  associated  with  any 
ATE  system,  any  one  of  which  represents  a  possible  source  of  ATE 
software  change.  They  are  the  user,  UUT  manager,  ATE  IM/SM,  ATE 
software  support  center,  and  data  automation.  The  user  refers  to  the 
actual  ATE  using  agency.  Depending  upon  the  level  of  testing  involved  the 
user  could  be  a  TRC,  a  TAC  maintenance  group,  or  one  of  many  other 
agencies.  TheUUT  manager  is  that  person  who  has  the  responsibility  for 
ensuring  that  the  UUT  is  kept  serviceable  and  operational.  This  person 
may  be  an  item  manager  or  a  system  manager  located  at  any  ALC.  An 
item  manager  or  system  manager  of  an  ATE  is  responsible  for  the  ATE 
elements  in  the  same  manner  as  is  the  UUT  manager  for  the  UUT.  Each 
ALC  and  AGMC  has  a  Software  Support  Center  (SSC)  assigned  to  the 
maintenance  directorate  to  act  as  the  focal  point  for  the  processing  of 
ATE  software  and  for  support  of  assigned  ATS.  Data  automation  provides 
support  for  off-line  Automatic  Data  Processing  Equipment  (ADPE)  such 
as  the  Remote  Job  Entry  Terminal  (RJET).  Table  3-4  indicates  an 
example  of  off-line  support  for  the  F-15  that  ATE  supplies  through  RJET 
from  Ogden. 

Any  of  the  aforementioned  activities  may  discover  the  need  for 
amending  ATE  software  and  initiate  processing  of  the  change.  In  addition, 
AGMC  has  a  software  support  center  which  maintains  an  organic  cap¬ 
ability  to  prepare  and  correct  calibration  test  software  used  to  calibrate 
precision  measuring  equipment 

3.3.2  Engine  and  Component  3  esting  Concept 

As  indicated  in  Section  1.4.2,  testing  of  engines  within  AFLC  is 
done  at  both  OC-Al.C  and  SA-AI.C,  Both  sites  use  the  Pacer  Comet 
system  to  test  engines  in  a  completely  assembled  condition  primarily  as 
a  result  of  returning  the  engines  to  the  depot  for  scheduled  maintenance 
or  overhaul.  The  overall  AFT.C  engine  test  workload  is  split  so  that 
certain  types  of  engines  are  tested  only  at  one  site;  e.g.,  the  F100  engine 
is  tested  at  SA-ALC  while  TF33  is  tested  at  OC-ALC.  If  an  engine  is 
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Table  3-4.  Off-Line  ATE  Software 
Support  Requirements 


Software  Element 

F-15  Adapted  PLACE  ATLAS 
(FAPA)  Compiler/Mini-FAPA 
Compiler 

MAGIC  BETA  Compiler 

MAC  Compiler 
LASAR  ATPG  (G2) 

ATPG  (TBD) 


Requirement 

LRU  test  program  generation  test 
procedures  manuals  status  log 
manipulation 

Module  test  program  generation 
compiler  change  support 

Module  test  program  generation 

Digital  test  pattern  generation 

Digital  test  pattern  generation 


I 
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disassembled  at  the  depot  (such  as  would  occur  during  overhaul)  the 
modules  or  components  of  the  engine  are  separately  tested  and  finally, 
upon  complete  engine  reassembly,  the  entire  engine  is  tested. 

Although  most  complete  and/or  component  engine  testing  is 
accomplished  at  the  depot  level  there  are  certain  tests  conducted  at 
field  levels.  Automatic  trim  testing  is  an  example  of  this.  All  engine 
test  set/operator  combinations  must  perform  certain  basic  tasks: 

•  Measure  engine  data 

•  Monitor  values  of  data  against  engine  safety  limits 

•  Reduce  the  data  to  a  useable  form,  i.e.,  engineering 
units 

•  Compute  standard  day  correction,  etc. 

•  Display  the  data 

•  Compare  the  displayed  values  against  trim  or  performance 
curves 

•  Determine  correction  action  for  out-of-limits 
conditions 

•  Record  data  and  test  results 

•  Provide  data  and  test  results  to  all  interested  agencies 

Engine  trim  systems  accomplish  part  of  all  of  these  tasks  (depending 
upon  trim  system  sophistication)  in  an  automatic  mode.  The  overall 
objective  of  trim  testing  is  to  minimize  cost  and  safety  risks.  This 
type  of  testing  can  be  envisioned  as  a  final  alignment  for  a  particular 
aircraft,  geographic  location,  and/or  mission  usage. 

Engine  component  testing  began  to  use  the  concept  of  "On-Condition 
Maintenance"  (OCM)  in  1977.  Primarily,  this  type  of  testing  uses  a 
building  block  approach  in  reassembly  of  the  engine  modules.  Modules 
are  broken  down  to  the  lowest  component  levels  and  the  components 
pooled  in  bins.  Reassembly  of  the  module  does  not  necessarily  use  the 
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same  combination  of  components  and  when  the  initial  components  are 
assembled  the  assembly  is  tested  at  this  level.  Satisfactory  testing 
means  that  more  components  can  be  added  and  then  at  that  level  be  tested 
"on  the  condition"  that  the  lower  subassembly  meets  testing  requirements. 
The  process  then  continues  until  the  engine  modules  are  completely 
assembled.  Several  portions  of  the  testing  are  controlled  automatically 
or  semi-automatically  with  computers. 

Interviews  with  maintenance  personnel  indicate  that  the  outcome 
of  using  these  engine  test  approaches  provides  very  favorable  results. 
Pacer  Comet  usage  sharply  reduces  the  number  of  testing  personnel 
from  a  comparable  manual  testing  level,  decreases  total  test  time,  and 
improves  fault  diagnosis  and  isolation.  Trim  testing  provides  increased 
assurance  that  the  engine  is  favorably  matched  to  its  environment  and 
thus  enhances  safety  considerations.  OCM  results  in  improved  testing 
results  and  decreased  testing  time  for  the  engine  modules. 

3.3.3  Change  Forms  and  Procedures 

Previous  discussion  indicated  why  ATE  software  often  needs  change 
and  the  likely  "discoverers"  for  any  changes  were  identified.  This  section 
addresses  the  mechanisms  by  which  changes  are  processed. 

Materiel  Deficiency  Reports  (MDR's)  provide  a  system  to  feed  back 
deficiency  data  on  hardware  and/or  computer  programs  to  activities  res¬ 
ponsible  for  the  development,  acquisition,  and  other  logistics  management 
functions.  MDR's  allow  action  to  be  taken  to  correct  and  prevent  materiel, 
design,  and  quality  deficiencies. 

There  are  two  types  of  MDR's,  Category  I  and  Category  II.  A 
Category  I  MDR  is  a  report  of  an  emergency  condition  on  all  types  of 
equipment,  systems,  and  computer  program  which  presents,  or  has  the 
clear  potential  to  present,  an  unacceptable  safety  hazard.  Computer 
programs  which  prevent  equipment  usage,  may  cause  Air  Force  mishaps, 
etc.  ,  can  be  reported  by  Category  I  MDR.  Category  II  MDR's  report 
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quality  deficiencies  in  materials.  Either  category  can  be  used  to  report 
an  ATE  deficiency  and  the  urgency  of  processing  the  change  dictates 
which  category  to  use.  Ultimately  the  MDR  is  sent  to  the  SM  or  IM  who 
is  responsible  for  the  computer  programs  and  documentation. 

AFR  800-14  and  associated  AFLC  supplements  require  computer 
program  deficiencies  be  identified  by  and  processed  as  MDR's  in 
accordance  with  Technical  Order  (T.O.)  00-35D-54.  This  supercedes 
use  of  both  AFTO  Form  22  and  Quality  Deficiency  Reports  (QDR's) 
although  AFSC  still  uses  QDR's.  Deficiencies  are  resolved  by  Materiel 
Improvement  Project  (MIP)  action  in  accordance  with  T.O.  00-5-15  and 
AFLCR  66-15.  Distribution  of  computer  program  changes  are  accom¬ 
plished  by  Time  Compliance  Technical  Order  (TCTO)  in  accordance 
with  T.O.  00-5-15,  AFR  57-4,  and  AFLCR  66-14. 

In  most  cases  MDR's  are  received,  processed,  and  distributed 
on  a  one-to-one  basis,  i.  e.  ,  one  MIP  and  one  TCTO  for  each  MDR 
received.  Each  ALC  has  a  focal  point  (usually  MMMS)  where  all 
externally  generated  MDR's  are  sent.  MMMS  screens  them  and  the 
ATE  MDR's  are  sent  to  the  appropriate  item  or  system  manager.  The 
flow  chart  in  Figure  3-4  is  indicative  of  the  r-mplex  administrative  data 
flow  the  MDR/MIP/TCTO  process  implements.  This  particular  flow 
exists  at  San  Antonio  ALC. 

It  is  important  to  know  that  UUT  software  or  test  program  MDR's 
can  be  submitted  from  either  the  organizational,  intermediate,  or  depot 
testing  levels.  UUT  MDR's  normally  precede  the  submission  of  an  ATE 
MDR  because  any  ATE  deficiency  is  likely  discovered  while  attempting 
to  test  or  diagnose  a  UUT.  Any  identified  ATE  deficiency  would  likely 
be  revealed  because  the  tester  is  not  portraying  an  accurate  analysis 
of  the  UUT  operational  state. 

3.3.4  Change  Process  Assessment 


Administratively  the  change  process  is  complex.  T.O.  00-35D-54, 
controls  the  way  that  ATE  software  changes  are  processed.  Command/ 
directorate  organizational  structures  and  mission  assignment  policies 
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cause  MDR's  to  oscillate  between  several  agencies  for  coordination 
purposes.  Each  oscillation  has  an  inherent  "human  factor"  delay  in 
addition  to  the  actual  work  time  for  the  document  coordination.  The 
cumulative  affect  of  these  delays  results  in  MDR's  with  simplified 
engineering  tasks  spanning  several  months  (difficult  tasks  take  longer) 
for  completion.  In  other  words,  adherence  to  T.O.  00-35D-54  and 
the  necessary  coordination  is  a  lengthy  process  and,  generally,  MDR 
processing  is  not  timely  with  respect  to  user  urgency. 

ATE  MDR's  are  initiated  to  correct  inadequacies  or  shortfalls  of 
the  ATE  testing.  Correction  of  the  inadequacy  is  not  always  straight¬ 
forward,  thus  the  associated  engineering  analysis  can  take  considerable 
time.  Although  the  responsibility  of  the  Engineering  Division,  the 
engineering  analysis  is  often  attacked  by  other  agencies.  The  coordi¬ 
nation  and  control  of  such  activities  are  not  efficient. 

3.4  ATE  ORGANIZATIONAL  CONCEPT 

As  mentioned  in  earlier  sections,  the  responsibility  for  ATE 
system  management  is  generally  assigned  to  SA-ALC  with  the  UUT 
assignments  being  made  to  IM's  at  the  ALC's.  The  resultant  organi¬ 
zations  are  aligned  to  support  the  ATE  assignment  responsibilities  in 
accordance  with  the  information  flow  as  assigned  by  AFLC  regulations 
and  as  depicted  by  the  diagram  in  Figure  3-5. 

The  ultimate  purpose  of  ATE  is  to  verify  the  operational  state  of 
a  UUT  so  in  most  cases  a  UUT  user  identifies  a  testing  need  for  a  UUT. 

He  submits  the  MDR  to  the  MDR  coordinator  at  the  ALC  who  has  UUT 
responsibility.  The  coordinator's  responsibilities  include  channeling 
MDR's  to  responsible  agencies  generally  identified  as  UUT  item  managers. 
Item  managers  generally  determine  that  each  MDR  requires  a  MIP  be 
established.  This  determination  may  require  a  hardware  and/or  soft¬ 
ware  engineering  analysis  and  in  some  cases  the  item  manager  can  send 
the  MDR  directly  to  the  Technical  Repair  Center  (TRC).  A  third 
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possibility  is  that  the  MDR  could  be  unnecessary  in  which  case  the  IM 
indicates  this  to  the  MDR  coordinator  and  all  actions  are  closed.  In  the 
event  the  engineering  analysis  requires  a  software  development,  the 
request  is  forwarded  to  the  software  developer.  Upon  completion  of  the 
software  development  the  results  are  included  in  the  ATE  testing  con¬ 
ducted  by  the  TRC.  Certain  ATE  requires  ADPE  support  and  additionally 
may  require  coordination  with  the  ATE  SM. 

It  has  been  previously  mentioned  that  MDR's  are  sometimes  written 
against  the  ATE  itself.  The  flow  is  approximately  the  same  except  the 
ATE  deficiency  will  likely  be  discovered  by  the  TRC  or  one  of  the 
engineering  analysis  groups. 

Specific  organizational  alignments  to  accomplish  the  aforementioned 
tasks  vary  from  ALC  to  ALC.  This  is  partially  because  the  assigned 
responsibilities  and/or  areas  of  emphasis  vary  for  each  ALC.  As  an 
example,  the  testing  associated  with  jet  engine  operation  is  quite 
different  from  testing  of  electronic  black  boxes. 

Typically  the  MDR  coordinator  is  located  in  MMMS  with  the  item 
managers  and  the  hardware  engineering  is  located  within  one  or  more 
item  divisions.  Software  engineering  analysis  is  located  within  the 
computer  resources  section  of  MME,  software  development  within  the 
software  support  center  of  the  maintenance  directorate,  the  TRC  functions 
within  the  maintenance  directorate,  the  ADPE  support  within  ACD,  and 
the  ATE  SM  is  located  at  San  Antonio  Air  Logistics  Center.  A  conception 
of  the  interface  complexity  of  these  agencies  can  be  gained  by  examina¬ 
tion,  from  an  organizational  viewpoint,  of  Figure  3-4. 

The  previous  few  paragraphs  and  Figure  3-5  are  presented  to 
indicate  the  basic  organizational  concept  involved  with  supporting  ATE 
software.  In  1975,  Software  Support  Centers  (SSC)  were  first  established 
to  play  an  important  role  in  this  concept.  Since  that  time,  several 
amplifying  regulations  and  letters  have  further  defined  the  SSC  and  its 
role.  Essentially,  SSC's  are  made  up  of  engineering  personnel  who 
use  ATE  equipment  controlled  by  Directorate  of  Maintenance  (DM) 
production  personnel  and  who  respond  to  Directorate  of  Materiel  Manage¬ 
ment  ATE  software  task  requests.  SSC  engineers  typically  support  D/MM 
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personnel  by  evaluating  ATS  deficiencies,  identifying  required  corrections, 
improving  ATE  software  efficiency  and  effectiveness,  reviewing  source 
data,  providing  preliminary  or  final  Form  1  drawings  of  the  ITA,  pro¬ 
viding  ATE  computer  program  documentation,  arranging  access  for 
computer  program  debug  and  verification/validation,  performing 
acceptance  testing  of  ATS  when  required  by  contract,  identifying  and 
projecting  requirements  for  off-line  ADPE  support,  and  providing  user 
and  programming  information  to  requesting  D/MM's  in  support  of  ATS 
acquisitions.  Additionally,  the  SSC's  support  other  maintenance  direc¬ 
torate  personnel  on  an  as-needed  basis  for  software  engineering  tasks. 

3.4.1  Organization  Concept  Assessment 

The  organization  associated  with  ATE  varies  from  ALC  to  ALC, 
thus  presenting  an  appearance  of  non- standardization  although  the  same 
regulations  apply  equally  to  each  ALC.  Figured -6  is  included  as  an 
example  using  only  the  SSC  at  each  ALC;  however,  similar  organizational 
dissimilarities  exist  within  the  Material  Management  Directorate.  This 
fact  coupled  with  interrelated  mission/item  assignments  cause  the 
organizations  to  be  complex.  As  the  complexity  has  increased  over  the 
years,  more  management  emphasis  has  been  applied  by  publishing  addi¬ 
tional  guidance  regulations.  In  many  cases  (see  Section  5)  these  regula¬ 
tions  overlap  so  that  some  activities,  such  as  engineering  analysis,  are 
redundantly  assigned.  The  result  is  that  the  current  organization  is 
complex,  difficult  to  manage,  and  somewhat  duplicative  in  responsibilities. 
In  spite  of  this  complexity,  the  people  are  making  the  concept  work. 

They  are  to  be  commended  for  their  activities,  dedication,  and  ingenuity. 
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WR-ALC  AGMC 


t  CONTAINS  THE  EQUIVALENT  OF  AN  SSC 


Figure  3-6.  AFLC  Software  Support  Center 
Chain  of  Command 


4.  REPRESENTATIVE  SYSTEM  AND  ATE  SUPPORT 


The  diversity  of  ATE  currently  existing  within  AFLC  is  enormous. 
Test  systems  which  automatically  respond  to  a  single  punched  card  input 
are  used  as  well  as  those  that  contain  sophisticated  computers  with 
multiple  operator  interfaces.  In  other  words,  there  is  a  wide  variance 
in  the  test  system  complexities.  It  has  already  been  emphasized  that 
digital/analog  testing  is  entirely  different  from  engine  testing.  This 
means  that  a  wide  variety  of  automatic  testing  applications  exists. 
Additionally,  three  testing  levels  of  depot,  intermediate,  organizational 
are  used  throughout  AFLC.  Many  of  the  weapon  systems  bear  little 
resemblance  to  each  other;  i.  e. ,  a  cruise  missile  has  little  in  common 
with  a  command  and  control  system  yet  both  use  embedded  computers 
and  both  require  ATE  support. 

In  view  of  these  considerations,  defining  an  approach  to  finding  a 
representative  sample  of  automatic  testing  support  problems  throughout 
the  command  presents  a  real  dilemma.  Upon  further  analysis,  it 
became  evident  that  if  testing  of  a  large,  complex  weapon  system  were 
considered  from  all  perspectives  then  most  of  the  ATE  support  problems 
would  be  addressed.  The  F-15  was  chosen  as  this  system  because  it  is  a 
sophisticated,  modern  system  that  makes  widespread  usage  of  automatic 
testing.  Accordingly,  that  system  is  presented  as  the  representative 
system  in  this  section.  This  does  not  mean  that  consideration  of  other 
weapon  systems  was  excluded. 

4.1  F-15  AUTOMATIC  TEST  EQUIPMENT  SUPPORT 

The  conglomerate  total  of  ATE  for  the  F-15  is  divided  into  the 
test  stations  shown  in  Table  4-1.  All  stations  currently  exist  except 
the  HP  ADTS  which  is  scheduled  for  delivery  commensuratewith  the 
programmable  signal  processor  update  to  the  APG-63  radar  set.  The 
test  station  applications  are  also  shown  in  the  table  and  are  self 
explanatory. 
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Table  4-2  depicts  an  overview  of  the  software  that  is  used  by  each 
F-15  ATE  station.  Table  4-3  summarizes  the  test  programs  that  apply 
to  each  test  station.  Table  4-4  summarizes  the  engine  test  stands  for 
testing  the  F100  engine  used  by  the  F-15  and  F-16  aircraft.  Table  4-5 
lists  the  associated  software  for  the  test  stands  for  the  F100  engine. 

The  composite  of  these  equipment  and  software  programs  makes 
up  the  support  baseline  for  the  F-15  weapon  system.  Figure  4-1  indicates 
the  interrelationship  of  all  F-15  ATE  systems  based  upon  a  breakout 
by  testing  level. 
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Table  4-3.  F-15  ATE  Test  Programs 


ATE  Statiori 

LRU/ 

Subassembly 

Module 

Engine 

Component 

AIS 

Computer 

17 

Displays 

10 

Microwave 

5 

ADTS 

Expanded  Memory 

Digital 

124 

Analog 

154 

IF  Video/Microwave 

28 

Multifunction 

51 

Comets  Digital-66FC 

31 

Comets-6861 

379 

AAI  5500 

1 

4 

INS  Station 

1 

31 

Accelerometer 

3 

Gyro 

2 

TEWS  Intermediate 

27 

TEWS  Depot 

100 

HP  ADTS 

Engine  CTE 

Unified  Fuel 

1 

Gas  Generator 

1 

Augmentor  Control 

1 

Augmentor  Spray  Manifold 

5 

Total 

61 

907 

8 
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5.  ASSESSMENTS  AND  DISCUSSIONS 


Automatic  testing  within  AKI.C  is  generally  being  satisfactorily 
accomplished;  however,  certain  inefficiencies  and  conflicting  manage¬ 
ment  controls  exist.  Testing  support  for  complex  systems  is,  like  the 
systems  themselves,  complex.  The  promise  of  increased  complexity 
is  apparent  for  the  near  future  as  appreciable  affects  of  any  acquisition 
standardization  are  not  expected  for  several  years.  Projects  such  as 
MATE  and  standard  data  buses,  as  well  as  better  up  front  planning, 
will  ultimately  enhance  testing  support  within  the  command. 

Overall,  the  current  assessment  of  ATS  within  the  command 
reveals  that  the  disciplines  and  controls  are  still  in  a  dynamic  state 
with  stability  not  yet  achieved.  The  Air  Force,  like  many  of  its  industrial 
counterparts,  is  evolving  an  improved  concept  of  ATE  management  which 
will  stabilize  within  the  next  few  years.  Recent  history  indicates  that 
some  weapon  systems  are  readily  automatically  tested  and  others  are 
still  striving  to  achieve  that  goal. 

The  following  paragraphs  are  offered  to  acquaint  the  reader  with 
some  of  the  evolving  areas.  Some  problems  and/or  issues  are  pointed 
out  in  the  discussion  and  should  be  considered  by  AFLC  Headquarters 
for  correction  or  improvement. 

5.1  ACQUISITION/DEVELOPMENT  DISCUSSION 

The  bulk  of  ATE  engineering  activities  occur  when  the  weapon 
systems  are  under  development.  Post-PMRT  activities  utilize  engineers 
for  both  hardware  and  software  changes;  however,  the  rates  of  change 
are  much  lower  than  during  the  full-fledged  development  period  so 
engineering  resource  utilization  is  lower.  Several  areas  of  the  following 
discussion  are  devoted  to  acquisition  and  development  because  it  is  this 
area  to  which  adjustments  can  be  made  to  provide  more  influence  upon 
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engineering  resources.  Implementation  of  the  areas  is  a  responsibility 
of  AFSC,  but  AFLC  inputs  arc  required.  Perhaps  a  bette_  under¬ 
standing  of  the  total  objective  of  the  acquisition  and  the  associated 
automatic  testing  will  improve  both  the  implementation  and  the  input 
data.  Specifically,  the  topics  to  be  discussed  are  the  Modular  Automatic 
Test  Equipment  (MATE)  project,  test  program  set  development,  UUT 
test  program  quality.  Design  For  Testability,  Software  Support  Center 
developments,  and  equipment  modifications.  There  is  some  degree 
of  overlap  in  these  various  subjects  because  they  influence  each  other, 
yet  there  is  a  distinctive  difference  in  the  various  subjects. 

5.1.1  Modular  Automatic  Test  Equipment  Project 

For  several  years  Department  of  Defense  expenditures  for  weapon 
system  maintenance  have  been  exceedingly  high.  Greater  than  20  percent 
of  the  annual  DOD  budget  is  spent  for  this  purpose.  Automatic  testing 
comprises  a  large  portion  of  the  maintenance  figure  and  proliferation 
of  ATS  is  widespread.  Accordingly,  the  Air  Force  has  initiated  a 
project  called  MATE  to: 

•  Develop  a  management  system  to  systematize  and 
control  the  acquisition  and  use  of  ATE 

•  Produce  a  broad  based  modular  hardware  and  software 
system  design  that  enables  multi-weapon  system  support 
at  all  maintenance  levels 

•  Generate  testability  criteria  to  be  applied  to  the  avionics 
and  ATE  designs 

The  MATE  concept  will  emphasize  centrally  coordinated  ATE  manage¬ 
ment  through  the  support  equipment  SPO  to  provide  interchangeable 
hardware  modules,  software  modules,  and  human  interface  modules  for 
various  ATE  systems.  USAF's  MATE  program  plan  is  depicted  in 
Figure  5-1 


5-2 


Currently,  the  MATE  project  is  in  a  competitive  environment 
between  Westinghouse  and  Sperry  both  preparing  the  guides  shown  in 
the  figure.  The  full-scale  development  will  be  awarded  after  the 
demonstration  phase. 

MATE  has  the  potential  for  lessening  the  AFLC  support  require¬ 
ments  for  ECS  software  because  of  the  applicability  of  modular  hard¬ 
ware  and  software  to  multiple  weapon  systems.  Additionally,  use  of 
standard  data  buses  such  as  IEEE  488,  1553B,  etc.  will  reduce  the 
equipment /software  proliferation. 

5.1.2  Test  Program  Set  Development 

A  Test  Program  Set  (TPS)  consists  of  a  complete  software  package, 
including  a  test  tape  or  disc,  interface  device,  and  test  program  instruc¬ 
tion.  The  complexity  of  the  TPS,  encompassing  the  ITA  hardware 
design  and  the  writing  of  the  test  program  (including  a  self-test  for  a 
complex  ITA),  is  one  of  the  major  costs  of  a  weapon  system  acquisition. 
The  degree  with  which  compatibility  is  attained  between  the  avionic 
hardware  (UUT)  and  the  ATE  impacts  the  complexity  of  the  ITA  and  test 
software.  When  testability  (see  paragraph  5.1.4)  is  designed  into  the 
hardware,  it  can  significantly  reduce  these  costs.  The  effectiveness  of 
ATE  support  is  directly  related  to  the  quality  of  the  TPS  and  the  design 
of  avionics. 

Figure  5-2  indicates  the  relative  costs  for  the  preparation  of  an 
ATE  computer  program.  Percentages  shown  are  approximations  based 
upon  data  compiled  for  previous  program  developments.  The  larger 
costs  are  associated  with  analysis  and  program  design.  Figure  5-3 
indicates  the  functions  involved  in  generating  TPS.  Considering  the 
activities  necessary  to  complete  the  generation  process,  it  is  no 
wonder  that  development  of  a  TPS  is  expensive. 

5.1.3  UUT  Test  Program  Quality 

As  mentioned  in  an  earlier  section  of  this  report,  many  of  the  UUT 
test  programs  are  delivered  to  AFLC  in  an  unuseable  or  incomplete 
state.  This  is  a  widespread  problem  which  has  several  factors  that 
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influence  the  UUT  test  program  quality,  thus  it  is  appropriate  that  tiu 
entire  process  of  conceiving  and  developing  test  programs  be  addressed. 

The  first  step  in  determining  what  ATE  and  test  software  is 
necessary  for  UUT's  is  to  conduct  a  level  of  repair  analysis.  This 
analysis  determines  and  compares  the  life  cycle  cost  for  repair  with 
the  life  cycle  cost  for  discard  at  a  given  maintenance  level,  justifies 
the  decision  to  repair  or  validates  use  of  ATE,  and  indicates  the  most 
powerful  variables  such  as  mean  time  between  failures  or  unit  cost. 

Step  two  is  a  test  requirements  analysis  of  a  UUT  to  determine 
performance  requirements  and  failure  modes  necessary  for  generating 
test  philosophy.  The  results  of  this  UUT  analysis  document  the  inter¬ 
face  between  UUT  design  and  test  program  design. 

Combining  steps  one  and  two  produces  the  Test  Requirements 
Document  (TRD).  A  TRD  aids  the  developer  in  standardizing  the  data 
collection  process  and  data  package  so  he  can  generate  a  test  program 
or  procedure  for  a  UUT  and/or  determine  the  type  of  equipment  required. 
Typically,  UUT  test  program  quality  is  directly  proportional  to  the 
TRD  quality  so  the  key  to  good  test  programs  is  good  TRD's. 

At  this  point,  an  analysis  needs  to  have  been  conducted  with  inputs 
from  AFLC,  AFSC,  and  the  UUT  vendor.  Note  also  that  TRD's  differ 
substantially  from  the  Part  I  or  II  Specifications  that  describe  the  UUT. 

A  mixing  of  the  objectives  of  specifications  with  the  objectives  of  TRD’s 
will  weaken  the  TRD.  Omission  of  a  good  input  from  either  of  the  three 
parties  will  also  weaken  the  TRD.  It  is  evident  that  "up  front"  planning 
and  analysis  is  the  real  key  to  effective  TRD's  and  the  resultant  UUT 
test  program.  Figures  5-4  and  5-5  are  included  to  indicate  the  functions 
involved  in  acquiring  TRD's  and  validating  the  TRD  data. 

Note  the  interfacing  of  TRD's  and  Test  Program  Set  (TPS)  as 
discussed  in  the  previous  section.  TPS  acquisition  is  a  big  cost  factor 
in  automatic  testing,  so  poor  quality  TRD's  actually  transfer  these  costs 
from  AFSC  to  AFLC  because  AFLC  has  been  completing  the  UUT  test 
program  development.  Summarily,  the  quality  of  TRD's  and  UUT  test 
programs  need  improvement. 
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Figure  5-4.  TRD  Acquisition 
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Figure  5-5.  TRD  Data  Validation 


5.1.4  Design  for  Testability 

Testability  of  UUT's  is  the  fundamental  reason  why  automatic  test 
equipment  are  procured.  Early  systems  (made  up  of  UUT's)  were  designed 
to  accomplish  a  function  or  group  of  functions  but  with  little  or  no  attention 
to  a  method  of  testing  the  components  of  that  system.  As  a  consequence, 
when  the  system  complexity  grew  to  an  extensive  state  some  of  the  system 
components  were  virtually  untestable.  Design  for  testability  then  is  a 
UUT  design  consideration  which  will  enable  the  UUT  to  be  tested.  [MIL- 
STD  2076  (AS)  addresses  UUT  compatibility  with  ATE  in  terms  of 
general  requirements]. 

On-line  testability  includes  Built-in-Test  (BIT)  and  is  an  inherent 
portion  of  the  normal  UUT  operation  even  though  the  UUT  may  only  be 
a  portion  of  a  system.  Enhanced  design  for  BIT  enables  the  UUT  or 
system  to  self  diagnose  or  at  least  to  "flag"  that  it  has  a  deficiency  or 
malfunction. 

In  design  for  testability  for  off-line  testing  the  objectives  are  to 
lessen  costs  through  better  field  performance.  Such  goals  as  easier 
test  design,  shorter  test  programs,  simpler  interface  devices,  and 
better  diagnostics  are  desirable.  In  some  cases  the  goal  might  be  to 
provide  test  points  for  electronic  probing  measurements. 

Design  for  testability  is  a  total  engineering  approach  combining 
the  activities  of  the  hardware  engineer,  software  engineer,  systems 
integrator,  and  an  integrated  logistics  system  engineer.  It  considers 
such  activities  as  fault  tolerance,  simulation,  life  cycle  costs,  standard 
hardware,  test  point  selection,  self-test  software,  BITE,  BIT,  failure 
modes,  accessibility,  partitioning,  and  redundancy.  (MIL-STD  2084 
addresses  modularity,  test  point  locations,  accessibility,  BIT,  and 
testability  verification). 
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Design  for  testability  is  recognized  as  a  fundamental  requirement 
which  must  be  levied  on  all  new  avionic  and  automatic  test  system  designs 
to  fulfill  MATE  goals.  The  basis  of  design-for-testability  is  a  MATE 
maintenance  concept  that  accounts  for  the  appropriate  support  trade¬ 
offs  and  maintenance  levels  required  by  the  target  weapon  system. 
Establishing  a  catalog  of  preferred  tests  is  key  to  standardizing  the 
approach  to  testability  designs.  Historical  information  and  improved 
design  approaches  are  being  developed  to  permit  effective  trade-off 
decisions  or  use  of  BIT/BITE  and  off-line  testing  and  calibration.  Both 
AFSC  and  AFLC  have  responsibility  to  ensure  that  analysis  of  technology 
trends  is  an  on-going  process  to  produce  systems  that  are  more  easily 
supportable. 

5.1.5  Software  Support  Center  Developments 

AFLCR  66-37  indicates  the  single  source  of  software  support  is 
the  ALC  Software  Support  Center  collocated  with  (or  determined  by) 
the  parameters  in  Table  5-1.  For  example,  if  the  ATE's  test  soft¬ 
ware  is  to  be  used  at  the  field  level,  then  the  SSC  should  be  collocated 
with  the  HUT  IM. 

AFLCR  23-4Z  paragraph  22b  states  the  SSC  "Designs,  develops 
and  provides  new,  altered,  updated  or  modified  test  software  and  updates/ 
corrects  existing  avionics  items/systems  software". 

This  statement  of  assignment  is  somewhat  impractical  because  some 
test  programs  such  as  built-in-test  arc  resident  on  the  operational  pro¬ 
gram  and  therefore  inseparable.  Other  test  programs  such  as  some 
field  test  programs  must  be  changed  and  distributed  with  the  operational 
software  change  and  it  is  impractical  to  have  an  ATE-SSC  located  at 
the  repair  source  make  the  actual  change. 

The  SSC  at  each  ALC  then  has  definite  responsibilities  for  software 
development  by  avionics  system  assignments.  Accordingly,  the  SSC  at 
each  ALC  is  manned  at  varied  levels  and  the  extent  of  involvement  in 
software  development  also  varies.  Table  5-2  summarizes  these  facts. 
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Software  Type 

ATE  Use 

Field  Level 
Only 

Depot  Level 
Only 

Field  and 
Depot 

Two  or 
More  Depots 

Test 

UUT-IM 

UUT 

ATE  User 

UUT-IM 

TRC 

TRC 

Self-T  est 

ATE-IM 

ATE  User 

ATE  User 

ATE-IM 

TRC 

TRC 

Control 

ATE-IM 

ATE  User 

ATE  User 

ATE-IM 

TRC 

TRC 

On-Line 

Support 

ATE-IM 

ATE-IM 

ATE-IM 

ATE-IM 
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Table  5-2.  SSC  Software  Development  Levels 


ALC  Location 

Approximate  SSC 
Manning  Level 

Approximate  Percentage 
of  Assigned  Personnel 
Engaged  in  Software 
Development 

Oklahoma  City 

45 

20 

Ogden 

68 

5  to  10 

Sacramento 

155 

70 

San  Antonio 

100 

15  to  20 

Warner  Robins 

140 

45 

The  percentages  in  Table  5-2  indicate  the  extent  of  involvement  by 
each  SSC  in  developing  new  software  for  agencies  outside  of  the  local 
TRC.  Any  software  modification  for  MDR's  on  UUT  test  programs, 
control  software,  or  support  software  is  not  included. 

Advantages  of  accomplishing  the  "outside"  development  are: 

•  In-house  expertise  is  enhanced  by  exposure  of  the  per¬ 
sonnel  to  actual  development  of  the  test  software 

•  Test  software  gets  quality  generated  (more  quality  and 
completion  problems  exist  if  the  UUT  vendor  develops 
the  test  software) 

•  Organic  development  appears  to  be  cheaper 

Disadvantages  of  accomplishing  the  "outside"  development  are: 

•  Other  software  support  may  be  impacted 

•  Application  of  AFLC  resources  to  an  AFSC  responsibility 

•  No  AFLC  mechanism  to  allow  AFLC  management  control 
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Analysis  of  the  advantages  and  disadvantages  will  lead  to  a 
conclusion  based  upon  the  perspective  of  the  analyzer.  In  other  words, 
if  one  wants  to  continue  outside  development  he  emphasizes  the  advan¬ 
tages.  Those  opposed  will  emphasize  the  disadvantages. 

There  are  two  primary  reasons  for  the  SSC's  using  "outside" 
development: 

•  AFSC's  history  of  delivering  quality  test  software 
is  poor,  so  a  "workaround"  was  implemented 

•  All  TRC  agencies /organizations  are  manned  based 
upon  their  productivity  with  "outside"  developments 
sometimes  undertaken  to  keep  the  people  busy,  thus 
precluding  any  personnel  reassignment  losses. 

5.1.6  Equipment  Modifications 

Class  IV  and  V  modifications  represent  a  large  portion  of  expendi¬ 
tures  to  maintain  weapon  systems  at  a  peak  operating  capability.  AFLC's 
expenditure  for  this  purpose  is  large  and  the  impact  upon  AFLC  budget 
and  resources  is  significant.  In  most  cases  a  change  to  a  weapon  system 
which  is  significant  enough  to  qualify  as  a  class  IV  or  V  modification 
will  necessitate  a  corresponding  change  in  the  ATE.  Sometimes  the  ATE 
change  can  be  accomplished  by  altering  existing  ATE  software,  but  in 
most  cases  there  is  need  to  change  the  ATE  hardware  or  perhaps  to 
acquire  additional  ATE  hardware.  Unless  adequate  up  front  planning 
and  consideration  for  the  affects  of  a  weapon  system  change  on  automatic 
testing  are  incorporated,  the  modified  weapon  system  is  likely  to  be 
unsatisfactorily  testable.  Any  of  the  factors  discussed  in  Sections  5,1.1 
through  5.1.5  can  be  involved.  AFLC  involvement  from  a  management 
pla  nning  perspective  should  be  as  thorough  for  a  modification  as  for  a 
new  system  acquisition. 

5.2  MANAGEMENT  CONSIDERATIONS 


Several  areas  are  included  as  a  subset  of  the  title  "Management 
Considerations".  They  are  presented  for  consideration  of  alternative 
approaches  that  correct  or  improve  existing  deficiencies.  None  are 
meant  to  be  of  a  demeaning  or  criticizing  nature  but  simply  to  indicate 
areas  where  improvement  is  needed. 


5.2.1  Agency  Responsibilities 

Regulations  which  address  ATE  acquisition  and  support  are 
numerous.  Many  are  complex  in  their  applications  and  interfaces  with 
other  documents.  Figure  5-6,  which  addresses  an  overview  of  AFR  300 
versus  AFR  800  acquisition  routes  for  ATE,  is  included  as  an  example. 
Although  acquisition  of  ATE  is  normally  an  AFSC  responsibility  it  is 
AFLC  who  is  left  with  the  responsibility  to  support  the  results  of  the 
acquisition.  Thus  if  the  acqusition  results  (weapon  systems)  are  poor, 
the  support  tasks  are  more  difficult. 

Assignment  of  responsibility  for  support  is  equally  as  complex 
as  the  acquisition.  Regulations  which  apply  are  numerous  but  generally 
ATE  is  covered  in  this  manner: 

•  DOD  Directive  5000.29  contains  the  basic  policy  under 
which  software  is  managed.  This  document  stresses 
system  management  rather  than  pure  software  manage¬ 
ment  and  it  establishes  ground  rules  for  life  cycle 
support.  It  requires  that  computer  programs  be 
managed  as  deliverable  configured  items. 

•  AFR  800-14  implements  the  broad  concepts  of  DODD 
5000.29  through  three  major  documents:  Computer 
Program  Development  Plan,  Computer  Resources 
Integrated  Support  Plan,  and  Operational /Support 
Configuration  Management  Procedures.  It  requires 
that  weapon  system  computer  programs  be  managed 
under  the  Computer  Program  Configuration  Item 
system  and  establishes  the  Computer  Program  Number¬ 
ing  (CPIN)  system.  AFR  800-14  further  established  the 
charter  for  organic  support  to  weapon  system  computer 
resources  to  include  ATE.  Volume  II  of  AFR  800-14 
provides  more  explicit  guidance  or  detailed  instructions. 
Among  these  instructions  are  such  items  as:  computer 
programs  are  to  be  managed  in  accordance  with  established 
configuration  management  practices,  computer  programs 
are  to  be  managed  as  part  of  the  system  to  which  they 
apply,  a  computer  program  change  is  a  modification  to 

the  system,  the  regulation  prohibits  the  split  responsibility 
assignment  for  hardware  and  its  associated  computer  pro¬ 
grams,  and  the  computer  program  change  process  is  an 
engineering  discipline  and  requires  system  integration. 
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Figure  5-6. 
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•  AFLC  Supplement  to  800-14  was  written  primarily  to 
interpret  for  ALCs  the  concepts  addressed  by  the  basic 
regulation. 

•  AF1.CR  800-21  was  written  to  provide  management  oriented 
guidance  that  further  amplified  AFR  800-14  guidance. 

•  AFLCR  523-1  established  the  policy  governing  mission 
assignments  to  all  AFLC  activities.  It  established  manage¬ 
ment  responsibility  for  ATE  hardware,  control  software, 
and  support  software. 

•  AFLCR  23-42  assigns  these  ATE  responsibilities  to  San 
Antonio  ALC.  It  also  outlines  the  SSC  responsibilities 
for  support  of  ATE  and  ATS. 

•  AFLCR  23  -43  outlines  the  materiel  management  responsi¬ 
bilities  for  ATE  engineering  support  and  analysis. 

•  AFLCR  66-27  assigns  tasks  for  automatic  data  processing 
in  providing  ATE  software  support. 

•  AFLCR  66-37  outlines  policies  for  ATS  management  and 
responsibilities  of  Headquarters  AFLC,  SM,  IM,  AGMC, 

ATE  users,  et  al.  for  implementing. 

•  Several  AFLC  Headquarters  letters  have  been  sent  over 
the  years  to  provide  guidance  to  the  SSC  and  engineering 
activities. 

•  AFLCR  400-1  outlines  guidance  to  the  Deputy  Program 
Manager  for  Logistics  (DPML)  who  resides  at  the 
acquisition  agency. 

Conflicts  in  assignment  responsibility  exist  because  of  redundant 
assignment  rather  than  omission  of  the  assignment.  For  example  AFLCR 
23-42  assigns  ATE  engineering  development  to  the  SSC  yet,  AFLCR  23-43 
assigns  engineering  analysis  to  the  Engineering  Division.  As  pertains  to 
changing  ATE  programs  the  terms  of  development  and  analysis  are 
virtually  synonymous.  Similar  redundancies  exist  in  other  areas. 

AFLCR  523  -1  policy  required  that  items  be  managed  by  FSCM's 
and  repaired  by  TRC's.  Exceptions  to  this  policy  have  been  approved 
and,  in  the  opinion  of  some  AI..C  personnel,  the  basic  policy  guidance 
has  been  seriously  eroded.  The  regulation  does  not  adequately  address 
such  items  as  LTUT  test  software,  FSC-70  manager  role,  and  built  -  in -test . 
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All  regulations  are  not  followed;  for  example,  in  its  truest  con¬ 
text  AFR  800-14  precludes  separation  of  hardware  and  software  engineers 
yet  at  every  ALC  the  ATE  hardware  engineers  are  assigned  to  the  item 
divisions  and  software  engineers  are  assigned  to  the  engineering  divisions. 
The  question  is  which  regulation  or  set  of  regulations  overrides  or 
supercedes  the  other  regulations. 

Most  ALC  personnel  feel  that  DPML's  are  not  truly  responsive 
to  ALC  inputs  for  ATE  acqusition  and  support.  These  feelings  are  based 
upon  the  remoteness  of  access  to  the  DPML's  and  the  fact  that  DPML's 
evaluation  reports  are  based  upon  inputs  from  acquisition  personnel 
who  possess  their  own  parochial  interests  that  do  not  enhance  AFLC 
objectives. 

Virtually  all  ATE  systems  contain  embedded  general  purpose 
computers  which  are  classed  as  FSG-70  equipment.  Yet,  the  FSG-70 
manager's  role  is  not  mentioned  in  any  of  the  AFLC  ATE  related  policy 
directives.  This  omission  results  in  vague,  incomplete  guidance  for 
the  management  of  the  computers  and  their  associated  control  and 
support  software. 

For  every  weapon  system  that  enters  the  Air  Force  inventory  there 
is  an  identified  control /focal  point  at  each  level  of  Air  Force  management; 
for  example,  SPO  Director,  program  element  monitor,  etc.  Subsystems 
such  as  avionics  or  landing  gear  do  not  have  a  separately  identified  focal 
point  yet  the  subsystems  are  an  integral  part  of  the  weapon  system  and 
thus  reflect  some  portion  of  the  weapon  system  capability.  Since  ATE  is 
not  an  operational  component  of  the  weapon  system  capability,  but  is  only 
a  support  tool,  the  impetus  to  acquire  the  ATE  is  less  than  for  the  weapon 
system  components.  Additional  impetus  could  be  provided  by  establish¬ 
ment  of  an  ATE  focal  point  at  Air  Force  Headquarters  level  similar  to 
that  established  for  embedded  computer  systems  or  the  electronic  war¬ 
fare  panel.  Some  method  of  centralizing  policy  and  procedural  guidance 
for  ATE  appears  to  be  needed  at  the  Air  Force  level. 
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5.2.2  Configuration  Management 

Originally  the  C PIN  system  was  conceived  as  an  alternate  method 
of  identifying  computer  programs  to  that  established  by  the  technical 
order  process.  The  main  objective  was  to  develop  a  system  that  pro¬ 
duced  more  configuration  control  and  that  uniquely  identified  each  com¬ 
puter  program  in  a  manner  that  traced  the  program  to  a  specific  weapon 
system  or  subsystem.  AFR  800-14  requires  that  each  computer  program 
be  controlled  at  computer  program  configuration  item  level.  This 
guidance,  coupled  with  CPIN  guidance,  can  allow  just  a  fewCPIN  numbers 
to  cover  the  multiplicity  of  software  programs  in  a  weapon  system.  The 
usefulness  of  this  system  has  not  been  demonstrated  at  the  ALC 
worker  levels. 

On  a  command-wide  basis  configuration  control  of  ATE  is  not  on 
a  quality  basis.  Weak  documentation  adds  to  this  problem,  but  mostly 
it  is  attributable  to  inadequate  policies  and  procedures.  Several  automated 
configuration  management  systems  have  been  attempted  yet  none  is  con¬ 
sidered  adequate.  On  a  lower  level  the  process  used  at  WR-ALC  seems 
to  adequately  address  this  issue  for  ATE;  however,  the  other  ALC's 
appear  to  be  in  a  weaker  environment.  This  assessment  is  made  because 
ATE  equipment  and  associated  programs  to  include  source  descriptions 
are  not  available  or  up  to  date  at  the  ALC's.  Configuration  control  implies 
that  not  only  are  compile  ATE  system  descriptions  known  and  available, 
but  that  changes  thereto  can  officially  be  made  only  after  adequate 
management  approval.  Several  areas  of  discrepancy  were  confirmed 
to  indicate  a  relative  lack  of  configuration  control. 

Software  changes  to  ATE  were  addressed  in  Section  3  and  are 
processed  via  MDR's  in  accordance  with  T.O.  00-35D-54.  Without 
exception  the  ALC  personnel  consider  this  process  too  lengthy  and 
suggest  that  it  be  streamlined. 
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5.2.3  Logistics  Management  Applied  to  Engineering  Problems 

Earlier  sections  of  this  report  presented  evidence  that  ATE 
support  has  become  an  engineering  problem  because  of  weapon  system 
and  tester  complexities.  Satisfactory  testing  hinges  upon  a  technical 
understanding  of  both  the  UUT  and  the  tester.  Item  and/or  system 
managers  are  rightfully  responsible  for  providing  management  of  UUT's. 
However,  in  some  cases  because  the  impacts  of  software  changes  to 
either  the  UUT  or  ATE  are  being  misjudged  by  the  responsible  managers 
and  because  requested  engineering  analysis  is  inadequate,  erroneous 
decisions  occur.  The  decisions  are  so  technical  and  complex  that 
meaningful  data  is  beyond  the  current  experience  level  of  some  item/ 
system  managers.  As  weapon  systems  increase  in  complexity  the 
frequency  of  complex  technical  decision  making  will  increase.  This 
means  that  either  the  managers  should  be  technically  trained  or  the 
engineering  decisions  should  be  delegated  to  the  engineers. 

Ironically,  a  similar  dilemma  extends  to  higher  management  levels. 
In  many  cases  the  mechanisms  do  not  exist  to  allow  meaningful  data  to 
get  to  the  ATE  decision  maker  in  a  clearly  understandable  state.  Thus 
the  decision  maker  faces  a  threat  of  acting  without  complete  knowledge 
of  the  impact  of  his  action. 

Coupled  with  the  management  concern  is  the  scarcity  of  travel 
funds  available  to  engineers  during  the  development  phases  of  weapon 
systems  and  their  associated  ATE.  This  scarcity  can  only  be  com¬ 
pensated  for  by  additional  "up  front"  planning  and  improved  quality 
from  the  acquisition  agency  with  additional  emphasis  applied  by  AFALD. 
Both  planning  and  acquisition  are  known  existent  problems  addressed 
in  Section  5. 1  of  this  report  so  rectification  will  not  be  easy.  In  many 
cases  the  absence  of  a  few  hundred  dollars  in  travel  money  results  in 
expenditures  of  additional  thousands  because  of  attempts  at  remote 
control  management. 
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5.3  INDUSTRY  VERSUS  AIR  FORCE  COMPARISONS 

Many  areas  of  concern  to  the  Air  Force  are  of  equal  concern  to 
industry.  For  example  "retest  O.K."  problems  are  just  as  prevalent 
in  industry  and  cause  just  as  many  reactions  as  in  the  Air  Force. 

Because  ATE  is  in  such  a  dynamic  state  and  many  of  the  same  problems 
exist  several  joint  actions  have  been  initiated.  Among  these  actions  are: 
the  Scientific  Advisory  Board  Panel  on  Automatic  Testing  currently  in 
session  and  expected  to  report  in  late  1980,  the  Industry /Joint/Services 
Automatic  Test  Project  begun  in  1977  and  completed  in  December  1979, 
and  several  groups  to  address  standardization,  modularity,  and  term 
definitions. 

The  results  of  the  Industry/Joint  Services  Automatic  Test  Project 
culminated  in  over  100  recommendations.  Summarization  of  the  system- 
software  development  and  maintenance  problem  is  quoted  from  the 
Executive  Summary  Report  as:  "The  development  of  automatic  test 
system  software  is  a  complex  process,  which  differs  subtly  from  that 
of  other  DOD  software.  Unfortunately,  there  is  no  consistent,  top- 
down  understanding  of  the  complex  hardware/software  relationships 
involved.  As  a  consequence,  cost  reduction  and  control  are  ineffective, 
and  software  maintenance  is  unnecessarily  hampered  by  the  many 
versions  of  non-rehostable,  proprietary  software  products  that  are 
developed.  " 

Project  recommendations  are  quoted  as:  "Rigorously  define 
software  life  cycle,  and  requirements  for  configuration  control  and 
quality  assurance"  and  "develop  guide  lines  for  configuration  management 
and  for  the  maintenance  of  automatic  test  system  software." 
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Those  recommendations  are  in  consonant  *'  with  this  study  effort 
although  they  more  aptly  address  AFSC  responsibilities.  AFLC  can 
improve  ATE  software  support  by  improving  management  personnel 
knowledge  of  ATE,  clarifying  assignment  responsibilities,  streamlining 
change  procedures,  and  applying  increased  emphasis  to  acquisition  of 
weapon  systems  and  test  systems. 

5.4  OTHER  ATE  INTEREST  AREAS 

Automatic  testing  concepts  were  initially  conceived  to  use  a  smart 
machine  that  could  be  operated  by  an  inexperienced  or  relatively  untrained 
individual.  As  the  concept  was  applied  to  the  support  of  the  F-lll  and 
the  F-15  the  discovery  was  made  that  the  ATE  did  not  reveal  the 
functional  faults  of  some  UUT's  that  were  known  to  contain  faults.  In 
fact  determination  of  the  faults  sometimes  presented  a  lengthy,  cumber¬ 
some  task  in  which  the  engineering  talent  was  more  or  less  pooled 
together  to  analyze  the  situation.  Additional  to  this,  the  Avionics 
Integration  Support  Facility  (AISF)  was  used  to  help  determine  and 
isolate  the  fault.  Typically  an  AISF  is  used  to  check  avionics  subsystems 
in  a  recreated  or  simulated  natural  environment.  These  elusive  faults 
seemed  to  lie  somewhere  between  a  pure  testing  scenario  and  an  AISF 
scenario.  Their  isolation  needed  a  more  analytical,  human  interactive 
approach  than  what  ATE  offered  but  less  than  a  full  AISF  environment. 

As  a  result,  some  "test  aids"  were  developed. 

5.4.1  Test  Aids 

Several  test  aids  are  in  various  stages  of  development  at  Sacramento 
and  at  least  one  is  envisioned  for  Warner  Robins.  These  aids  and  the 
need  for  them  were  locally  determined  at  the  individual  ALC  and  are 


either  being  developed  or  will  be  developed  by  that  ALC.  Development 
of  ideas  such  as  these  are  a  reflection  of  the  attitude  of  the  ALC  per¬ 
sonnel  to  get  the  job  done. 

The  following  test  aids  are  in  development  at  SM-ALC. 

•  Flight  Line  Computer  Loader  (FLCL)-F-111D/F/FB. 

New  capability  to  load  and  verify  General  Nav  Computer 
(GNC);  Weapons  Delivery  Computer  (WDC);  Navigation 
Computer  Unit  (NCU);  and  SRAM  OFP  at  the  flight  line. 

Also  provide  avionics  diagnostics  at  flight  line. 

•  Mission  Data  Terminal  (MDT)  -  F-111D/F/FB. 

Replace  existing  mission  tape  preparation  unit  (AGERD 
6660)  with  new  equipment  capable  of  providing  both 
punched  and  magnetic  tape  containing  mission  data. 

•  Mission  Data  Loader  (MDL)  -  F-111D/F/FB.  Replace 
existing  punched  tape  mission  data  loader  (AGERD  6659) 
with  new  equipment  to  load  mission  data  from  magnetic 
tapes  through  the  wheel  well  fill  into  the  aircraft 
computers. 

•  Digital  Computer  Complex  (DCC)  Test  Aid  (DCC-TA)  - 
F-111D/F/FB.  Provide  capability  to  perform  dynamic 
system  test  at  field  shop.  This  will  provide  a  new  capa¬ 
bility  in  field  shop  to  identify  and  fault  isolate  system  tape 
time  dependent  and  intermittant  problems.  This  is  beyond 
capability  of  existing  field  ATE.  DCC  includes  GNC /WDC 
computers  and  converter  set. 

•  Inertial  Navigation  Set  (INS)  Test  Aid  (INS-TA)  - 
F-lllD/F/FB.  Provide  capability  to  perform  dynamic 
system  test  at  field  shop.  This  will  provide  a  new  capa¬ 
bility  in  field  shop  to  identify  and  fault  isolate  system  type 
time  dependent  and  intermittant  problems.  This  is  beyond 
capability  of  existing  field  ATE,  INS  includes  Navigation 
Computer  Unit  (NCU);  Inertial  Reference  Unit  (IRU);  and 
Battery  Unit  (BU). 

•  Inertial  Navigation  Set  (AN/AJQ-20)  Test  Aid  -  F/1UA. 
Provide  capability  to  perform  dynamic  system  test  at 
field  shop.  Provide  same  capability  for  F-111A  as  INS- 
TA  for  F/111D/F/FB  described  above;  except  this  test 
aid  will  be  manually  operated.  The  AN/AJQ-20  is  an 
analog  system,  no  digital  computers.  System  includes  a 
Stable  Platform  Unit  (SPU)  and  an  analog  computer. 


•  Automated  Documentation  Tool  for  Use  With  F-1HD/F/FB 
OFP  Development  in  AISF.  Provide  a  means  to  automatically 
generate  software  flow-charts.  Considerable  savings  on 
flow  time  and  drafting  effort. 

•  Digital  Signal  Transfer  Unit  (D-STU)  Data  Analyzer. 
Microprocessor  controlled  data  analyzer  for  testing  new 
D-STU  used  in  F-111D  only. 

•  Navigation  Computer  Unit  (NCU)  Memory  Modification  - 
F-111D/F/FZ*.  Modification  to  improve  reliability  of  NCU 
by  replacing  existing  core  memory  with  Programmable 
Read  Only  Memory  (PROM)  and  improved  timing.  This 
will  preclude  memory  scrambles  in  NCU  and  improve 
reliability.  Flight  testing  by  SAC  (several  thousand  hours) 
shows  an  MTBR  improvement  form  approximately  45  hours 
to  over  300  hours. 

The  following  test  aid  is  programmed  at  Warner  Robins.  The 
System  Test  Analysis  Center  (STAC)  will  be  designed  to  resolve  all 
test  void  and  retest  O.K.  problems  within  the  F-15  maintenance  system 
at  all  levels  of  maintenance.  It  will  be  established  in  accordance  with 
the  requirements  set  forth  in  the  F-15  ATE  CRISP  (Automatic  Test 
Equipment  Computer  Resources  Integrated  Support  Plan). 

5.4.2  Test  Approach  to  F-16  ATE 

In  determining  the  ATE  needed  to  provide  avionics  test  support  to 
the  F-16  all  of  the  software  support  requirements  were  gathered 
together  and  grouped  according  to  basic  functional  tests.  This  produced 
a  four-station  test  approach.  Each  station  design  was  analyzed  to  deter¬ 
mine  what  common  equipment  could  suffice  in  each  test  station.  As  a 
result  a  "common  core"  was  developed  which  means  that  interfaces  to 
the  core  are  compatible  from  test  station  to  test  station.  Further  the 
components  of  the  core  are  interchangeable.  Subsequent  to  completion 
of  the  prototype  test  stations  the  testers  were  fielded  to  future  users 
but  with  vendor  maintenance  and  operation  personnel  to  support  the 
testers.  During  this  subsequent  time  the  users  used  on-the-job- 
training  and  were  able  to  suggest  enhancements  for  the  tester  man- 
machine  interface. 


In  summary,  the  tester  design  was  "front  end"  planned  with 
standardization  and  equipment  interchangeability  in  mind  where 
practical.  The  testers  were  fielded  for  trial  usage  where  operational 
improvements  were  made.  The  results  look  very  promising. 

This  approach  appears  to  already  have  demonstrated  some  of 
the  objectives  of  the  MATE  program. 
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APPENDIX  A 

SA-ALC  ATE  EQUIPMENT  TEST 
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